Deep learning-based image augmentation for additive manufacturing

Abstract

A method for enhancing image resolution for a sequence of 2D images of additively manufactured products is provided. For each of a plurality of additive manufacturing processes, the process obtains a plurality of successive low-resolution 2D images of each of the respective products during the respective additive manufacturing processes and obtains a respective high-resolution 3D image of each of the respective products after completion of the respective additive manufacturing processes. The process selects a tile placement map that subdivides the low-resolution 2D images and the high-resolution 3D images into low-resolution tiles and high-resolution tiles, respectively. The process also iteratively constructs an image augmentation generator in a generative adversarial network using training inputs including ordered pairs of low-resolution and high-resolution tiles. The process stores the image augmentation generator for later use to augment the sequence of low-resolution 2D images captured of the product during additive manufacturing.

Description

Technical Field

[0001] The disclosed implementations generally relate to additive manufacturing, and more specifically, to systems, methods, and user interfaces for deep learning-based image enhancement for additive manufacturing.

Background Art

[0002] Due to the complexity of additively manufactured parts (e.g., parts manufactured using 3D printing), non-destructive quality control methods are very limited. The most widely used non-destructive testing method is micro-CT (computed tomography) scanning. This process achieves higher geometric accuracy, but this process is extremely expensive and time-consuming and does not scale to large parts made from high-density materials (e.g., this process is affected by shadowing artifacts in high-density materials). Some systems use digital twin 3D volumes based on near-infrared (NIR) images. Although excellent pore or crack sharpness is achieved by NIR post-processing, inaccurate geometric dimensions & tolerances (GD&T) result, and post-build effects are not predicted. The 3D volume is also limited by the resolution of the image. Furthermore, such systems only capture data during the manufacturing process, which means that subsequent changes are not captured. For example, these systems do not capture the remelting of metal that changes the final geometry.

Summary of the Invention

Means for Solving the Problems

[0003] In addition to the problems described in the background section, there are other reasons why an improved system and method for inspecting additive manufacturing quality are needed. For example, existing techniques rely on post-analysis of additively manufactured products, so there is a lack of situational information for proper root cause analysis. The present disclosure describes systems and methods that address the drawbacks of conventional methods and systems.

[0004] This disclosure describes a system and method that addresses some of the shortcomings of conventional methods and systems. This disclosure uses a deep neural network technique called a Generative Adversarial Network (GAN) to simultaneously improve the resolution of NIR images, model post-construction effects (such as remelting or shrinking), and convert the data into a format usable by commercially available CT analysis tools. The techniques described herein enable the neural network to process large 3D volumes. This disclosure describes two separate processing pipelines. The first pipeline is used for training and testing (an ongoing process to improve the quality of the algorithm), and the second pipeline is used for field deployment.

[0005] This disclosure describes how to construct digital models for on-site inspection of additive manufacturing quality using computer vision, machine learning, and / or statistical modeling, according to several implementation forms. The techniques described herein can be used to extend low-resolution NIR images to CT-like quality and resolution. In addition, the output can be analyzed with commercially available CT scan software. The techniques incur very little cost per build for highly accurate comprehensive GD&T. The techniques improve feature detection for features such as cracks or pores. The techniques can also be used to predict post-layer effects such as remelting, expansion, and contraction, based on training from previous builds. Unlike CT, systems following the techniques described herein do not generate CT-like scan artifacts and help reduce noise in the output.

[0006] In some implementations, the present invention uses one or more cameras as sensors to capture a series of images (e.g., still images or video) during the additive manufacturing process of a product. The temporally continuous images are processed as a multidimensional data array using computer vision and machine / deep learning techniques to perform relevant analysis and / or predictive insights. This includes steps of locating specific features (e.g., defects) and determining the degree of those features to assess quality.

[0007] In some implementations, images of the ongoing additive manufacturing process are processed using trained computer vision and machine / deep learning algorithms to characterize defects. In other implementations, computer vision and machine / deep learning algorithms are trained to determine product quality based on images of the ongoing additive manufacturing process.

[0008] According to some implementations, the method runs on a computing system. Typically, a computing system includes a single computer or workstation or multiple computers, each having one or more CPU and / or GPU processors and memory. The machine learning modeling methods implemented generally do not require a computing cluster or supercomputer.

[0009] In some implementations, a computing system includes one or more computers. Each computer includes one or more processors and memory. The memory stores one or more programs configured for execution by one or more processors. One or more programs include instructions for performing any of the methods described herein.

[0010] In some implementations, a non-temporary computer-readable storage medium stores one or more programs configured to be executed by a computing system having one or more computers, each having one or more processors and memory. The one or more programs include instructions for performing any of the methods described herein.

[0011] Thus, methods and systems for facilitating on-site inspection of additive manufacturing processes are disclosed. The discussions, examples, principles, compositions, structures, features, arrangements, and processes described herein can be applied, adapted, and embodied in additive manufacturing processes.

[0012] To better understand the disclosed systems and methods, as well as any additional systems and methods, refer to the following drawings along with the descriptions of implementations below, where similar reference numbers refer to corresponding parts throughout the drawings. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram of a system for on-site inspection of additive manufacturing processes, according to several implementation configurations. [Figure 2A] This is a block diagram of a system for training a Generative Adversarial Network (GAN) to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementation configurations. [Figure 2B] Figure 2A shows block diagrams of generative adversarial networks (GANs) according to several implementation configurations. [Figure 2C] This is a block diagram of a system that uses a Generative Adversarial Network (GAN) to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementation configurations. [Figure 3] This is a block diagram of a computing device according to several implementation configurations. [Figure 4] This is a diagram illustrating the layers of an additive manufacturing process, following several implementation configurations. [Figure 5] This diagram illustrates the steps involved in stitching together image tiles for additive manufacturing processes, according to several implementation configurations. [Figure 6A] This figure shows wall thicknesses measured using NIR, micro-CT scans, and artificial high-resolution imaging, according to several implementation configurations. [Figure 6B] This table shows example measurements based on geometric comparative tests according to several implementation configurations. [Figure 7] This is an illustrative visualization showing wall thickness measurements according to several implementation configurations. [Figure 8A] This figure shows examples of artificial high-resolution images, following several implementation configurations. [Figure 8B] This figure shows examples of artificial high-resolution images, following several implementation configurations. [Figure 9] This is a schematic diagram of a method for Z-axis enhancement (interpolation) according to several implementation forms. [Figure 10] This block diagram illustrates a system for training a generative adversarial network to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementation configurations. [Figure 11] This block diagram illustrates a system that uses a generative adversarial network to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementation configurations. [Modes for carrying out the invention]

[0014] References to implementation forms are made here, and examples are illustrated in the attached drawings. The following description includes many specific details to provide a complete understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be carried out without requiring these specific details.

[0015] Figure 1 is a block diagram of a system 100 for on-site inspection of an additive manufacturing process according to some implementation forms. The additive manufacturing apparatus 102 is monitored by one or more camera devices 104, and each device 104 includes one or more image sensors 106 and one or more image processors 108. The data collected by the camera devices is communicated to the on-site inspection server 112 using the communication network 110. The additive manufacturing apparatus 102 uses a set of additive manufacturing parameters 118, which can be dynamically updated by the on-site inspection server 112. The manufacturing parameters can include a detailed process flow that defines both the materials used and how the process is implemented.

[0016] The on-site inspection server 112 uses some standard computer vision processing algorithms 114, as well as some machine / deep learning data models 116.

[0017] During the additive manufacturing operation, the process captures images on-site and applies standard image processing techniques to enhance features (e.g., Gaussian blur, edge detection of electrodes and molten pools, signal-to-noise filtering, and angle correction). This process uses temporal cross-correlation to align image stacks or video frames geometrically. In some implementation forms, this information is supplied to one or more mounted robotic cameras for accurate image capture. The system converts the temporal image trend into a stationary signal by taking the temporal derivative of the image. The system trains a convolutional neural network on consecutive and delayed image batches with 3D convolution (e.g., pixel position, intensity, and color / spectral band). In some implementation forms, the machine / deep learning data model 116 outputs the probability of certain types of events (e.g., yes / no, or one of the defect types).

[0018] Figure 2A is a block diagram of a system 200 for training a generative adversarial network (GAN) to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementations. Raw input images 202 (e.g., NIR images or CT data) are obtained from on-site monitoring of the additive manufacturing process. Some implementations obtain initial data (raw input images 202) by capturing NIR images (while the product is being built) at a predetermined resolution (e.g., 60 μm (micrometer) resolution). Some implementations perform micro-CT scans of printed portions or products at a second predetermined resolution (e.g., 20 μm resolution). Micro-CT scans are typically generated after the product is fully built. In some implementations, each NIR image corresponds to an XY cross-section (e.g., layer) of the product. The input images are cropped and aligned to obtain NIR and CT data 204, which are then tiled with curvature deformation (e.g., random xy, yz, or xz tiles) to obtain intermediate tiled input 206. A stack of these tiles (each representing a portion of the overall image) is used to train a generative adversarial network 208. For example, each NIR image corresponds to an XY cross-section or layer of the product, and the CT image stack contains scans of the product from bottom to top layers (i.e., the bottom layer is first, followed by subsequent layers, all the way up to the top layer). The step of aligning the NIR and CT scan images involves matching the images layer by layer, from the bottom layer to the top layer. Some implementations augment the input data (the data used to train the neural network) by modifying the input data to increase the amount of data available for training. Some implementations rotate, reposition, and / or scale the input data to obtain more data for training. In some cases, these actions do not actually provide additional data. For example, rotation does not provide additional data or information when the input data or image contains circles or hexagons. Some implementations augment the data by curving the data.In some cases, the curvature generates additional information (e.g., H-shaped) used to train the neural network. The curvature is used to augment the training data set to increase the diversity in the training data. The curvature (often called perspective warp) involves an affine transformation of the input data. Some implementations warp NIR images (or tiles obtained from NIR images) and CT images (or tiles obtained from CT images) to generate corresponding warped images (or tiles) for training. The warping operation takes a 2D image and projects it into 3D space. This causes the image to be displayed as if viewed at an angle. Since the final image is reprojected into 2D, the curvature generates an image that appears stretched, twisted, bent, or deformed in other ways. Some implementations perform the warping operation equally on both the CT input data and the NIR input data, generating a ground truth pair for geometric shapes or features not seen in the original data.

[0019] Some implementations pair the data sources by assembling NIR images into a 3D volume, aligning the NIR volume and the micro-CT volume, and / or upscaling the NIR images to the CT resolution (e.g., using basic interpolation). Some implementations extract the training set by randomly selecting tiles from the paired 3D volumes described above, randomly manipulating the data to augment the data set, and / or splitting the data set into training, test, and validation sets of data. Some implementations then use the training set to train the GAN208.

[0020] Figure 2B is a block diagram of the Generative Adversarial Network (GAN) 208 shown in Figure 2A, following several implementation configurations. Generative adversarial networks, or GANs, are advanced methods for training deep convolutional neural networks (CNNs). Instead of using a single network, two separate networks (generator 212 and discriminator 216) are trained. The generator 212 is the network that will ultimately be used after training. The generator 212 takes input data 210 (e.g., near-infrared or NIR images), processes the input data 210, and generates a sample 218 (e.g., fake or artificial CT data). The result 218 is considered "fake" data because it is generated (and not an actual CT scan). The discriminator 216 takes the original "true" sample 214 or the "fake" generated sample 218 (or both simultaneously) and attempts to determine whether the data is "true" or "fake". The two networks are trained simultaneously. The generator 212 is trained on its ability to "deceive" the discriminator 216 into believing that the data is "true," while the discriminator 216 is trained on its ability to distinguish "true" from "false." This causes both models or networks to become increasingly accurate. Once the accuracy of both networks stabilizes (e.g., accuracy does not change over several iterations), the models are considered "converged" or "fully trained" (a state shown by block 220). In some cases, convergence occurs when the discriminator IC corrects for about 50% of the time. The state shown in block 220 provides fine-tuning feedback 222 to the generator 212 and discriminator 216 until the network converges. In some implementations, a human developer verifies the quality of the "false" data results. During the GAN training process, the network uses the training dataset (described above) to train the network and an independent validation dataset (also described above) to measure its accuracy. The trained model generates statistics (e.g., accuracy and / or loss) for each input.In some implementations, once the training reaches a point where the developers consider the output quality to be good, testing is performed using a completely different dataset. This dataset is essentially a rehearsal for the product model, testing not only the neural network itself but also the entire image processing and assembly pipeline.

[0021] Figure 2C is a block diagram of a system 224 for using a generative adversarial network (GAN) 208 to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementations. The raw input image 226 (e.g., NIR image) is obtained from on-site monitoring of the additive manufacturing process of the product. Some implementations capture layered images of the product over the additive manufacturing process period. Some implementations change the size of the image (e.g., triple the size). The input image is cropped to obtain a cropped NIR image 228. Some implementations segment the cropped image 228 into numbered tiles (e.g., 256x256 pixel tiles) and extract tiles for each predetermined number of pixels (e.g., every 128 pixels). In some implementations, the tiles overlap (e.g., four tiles overlap) to eliminate edge effects. The cropped NIR image is then ordered and / or tiled to obtain ordered tiles 230. The ordered tiles 230 are input to a GAN generator 208 being trained (operating in estimation mode) to obtain the ordered tile output 232. The ordered tile output 232 is stitched and mixed to obtain output 234. Output 234 is used to reconstruct the digital twin model 236 (for example, the stitched images are attached to form a 3D volume). Details of the tile arrangement and the algorithm for stitching the images, according to several implementations, are described below with reference to Figures 4 and 5. In some implementations, each tile is input to the GAN separately. The input image is sliced ​​into tiles, each tile is processed by the GAN, and then the output tiles are reassembled into the final image. This is done for each image in the image stack (of tiles), and finally the stack of output images is combined into a 3D volume.

[0022] Figure 3 is a block diagram illustrating a computing device 300 in several implementation configurations. Various examples of computing devices 300 include servers, supercomputers, desktop computers, cloud servers, and high-performance clusters (HPCs) of other computing devices. The computing device 300 typically includes one or more processing units / cores (CPU and / or GPUs) 302 for executing modules, programs, and / or instructions stored in memory 314 and thereby performing processing operations, one or more network or other communication interfaces 304, memory 314, and one or more communication buses 312 for interconnecting these components. The communication buses 312 may include circuits for interconnecting and controlling communication between system components.

[0023] The computing device 300 may include a user interface 306 comprising a display device 308 and one or more input devices or mechanisms 310. In some implementations, the input device / mechanism includes a keyboard. In some implementations, the input device / mechanism includes a “soft” keyboard, which is displayed on the display device 308 as needed, allowing the user to “press keys” that appear on the display 308. In some implementations, the display 308 and the input device / mechanism 310 comprise a touchscreen display (also called a touch sensor display).

[0024] In some implementations, memory 314 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, memory 314 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. In some implementations, memory 314 includes one or more storage devices located away from the GPU / CPU 302. Memory 314, or alternatively, the non-volatile memory devices within memory 314, comprises a non-temporary computer-readable storage medium. In some implementations, memory 314, or the computer-readable storage medium of memory 314, stores the following programs, modules, and data structures, or subsets thereof: • Operating System 316, which handles various basic system services and includes procedures for performing hardware-dependent tasks. • One or more communication network interfaces 304 (wired or wireless), and a communication module 318 used to connect the computing device 300 to other computers and devices via one or more communication networks such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. • An optional data visualization application or module 320 for visualizing and displaying defects added during on-site inspection. • Input / output user interface processing module 346, which allows the user to specify parameters or control variables. • Field inspection engine 112 (described above with reference to Figure 1, often referred to as the inline monitoring engine). In some implementations, the inspection engine 112 includes an image processing module 322 and / or an additive manufacturing control module 330, as described below with reference to Figure 10. In some implementations, the image processing module 322 includes an image tile placement module 324, an image stitching module 326, and / or an image transformation module 328, as described in more detail below. - A tile placement map 332, such as the one used by the image tile placement module 324, to subdivide a low-resolution 2D image to obtain low-resolution (LR) tiles, and / or to subdivide a high-resolution 2D image to obtain high-resolution (HR) tiles. • Machine / deep learning / regression models 334 (for example, model 116, as further described in and below Figure 1), including weights and input vectors. • Additive manufacturing process 336, • 338 consecutive low-resolution 2D images • 340 high-resolution 3D images, including slices of high-resolution 2D images. Optionally, build post-effect model 342, and / or • Interpolation module 344

[0025] Each of the executable modules, applications, or sets of procedures identified above can be stored in one or more of the previously mentioned memory devices and corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) identified above do not need to be implemented as separate software programs, procedures, or modules, and thus can be combined into various subsets of these modules, or otherwise rearranged in various implementation forms. In some implementation forms, memory 314 stores a subset of the modules and data structures identified above. Furthermore, memory 314 can store additional modules or data structures not described above. The operation and data structure properties of each of the modules shown in Figure 3, according to some implementation forms, are described further below.

[0026] Figure 3 shows a computing device 300, but it is intended not as a schematic representation of the structure of the implementation described herein, but rather as a functional description of the various features that may be present. In practice, as will be recognized by those skilled in the art, the items shown separately can be combined, and some items can be separated.

[0027] In some implementations, though not shown, memory 314 also includes modules for training and running the models described above with reference to Figures 1 and 2A-2C. Specifically, in some implementations, memory 314 also includes a probabilistic sampling module, a coding framework, one or more convolutional neural networks, a statistical support package, and / or other images, signals, or related data.

[0028] Figure 4 is an illustrative tile diagram of layers of an additive manufacturing process according to several implementations. Images 400, 402, 404, and 406 can correspond to different layers of the same product (or different products built using the same or different additive manufacturing processes) according to several implementations. According to several implementations, each image is tiled according to a tile size that is a fraction of the image size (e.g., 1 / 4 or 1 / 8 of the image size). Some implementations use the same tile size to tile all images. Some implementations use different tile sizes for different products and / or different additive manufacturing processes. In Figure 4, images 400, 402, 404, and 406 correspond to separate sets of tiles from the same image. Some implementations combine these sets of tiles to create sets of tiles that overlap in each direction. For example, a high-resolution image typically contains a total of 400 tiles (4*100), with 10 or more tiles on each axis. The tiles are offset, and some tiles have edges that are half the thickness in one or more directions. In some implementations, the GAN requires tiles to be of uniform size, and therefore these tiles are not included in the computation (e.g., the cut-off portion is filled with zeros), or the cut-off portion is mirrored from the uncut portion.

[0029] Figure 5 shows the stitching steps of layered image tiles in an additive manufacturing process according to several implementations. Some implementations use an image stitching algorithm that achieves superior blending of tiled images compared to conventional methods where detail is not lost. Each pixel in the output image is generated by the weighted sum of its value in each of four or more overlapping tiles. The weight of each tile is linearly proportional to the distance from the center of the tile to eliminate edge effects. Figure 5 shows an initial tiled image 502 that can be processed by the stitching algorithm 500 to create a stitched tiled image according to several implementations. In Figure 5, image 500 shows overlapping tiles, with edges from each overlapping tile appearing in the image. Some implementations generate image 500 by stitching four overlapping tile configurations and blending them together by considering representative or average values ​​of the four images (each image corresponding to an overlapping tile). Figure 5 also shows a second image 502 generated using the improved blending method. In some implementations, before stitching each set of tiles together, the tiles are multiplied by a "mixed tile" generated by a mathematical formula. An example of the formula is shown below.

[0030]

number

[0031] In equation (1) shown above, x and y are the x,y coordinates (of the image pixels), and width and height are the width and height (dimensions) of the tile corresponding to the pixel. In some implementations, equation (1) generates a weighted image with a scale from 0 to 1, determined by the x and y distances from the center of the image. In some implementations, the images generated by these four configurations are added together such that the sum of the weights of each pixel is 1.

[0032] Figures 6A and 6B show the results of testing the output of a trained neural network (e.g., GAN208) according to several implementations. The test results are based on caliper measurements for NIR, CT, and pseudo-CT (artificial images created by the GAN208 generator) according to several implementations. The test results are from geometric comparison tests, which include calibrating pixel size by measuring the full width of a portion at the pixel level, measuring the thickness of thin walls (the smallest possible features) using calipers, and comparing the measurements for the system's output with the wall thickness measured for NIR and micro-CT scans, according to several implementations. Figure 6A shows a cross-section of an additively fabricated portion measured in the width direction using calipers, according to several implementations. The width measurement is divided into the average pixel width of this region in each image to determine the voxel size. By utilizing the full width of the portion, the measurement error ratio due to the uncertainty of the pixel edge boundary by + / -1 pixels is minimized to the fraction + / -2 / (total number of pixels). In Figure 6A, the first image 600 is an NIR image (a low-resolution input image, including a minimum of 427 pixels, an average of 228 pixels, and a maximum of 431 pixels). The second image 602 corresponds to a CT scan image (often referred to as a CT image, which is a high-resolution image including a minimum of 1367 pixels, a maximum of 1373 pixels, and an average of 1369 pixels). The third image 604 corresponds to the output generated by the system (a fake CT image, which includes a minimum of 1294 pixels, a maximum of 1301 pixels, and an average of 1297 pixels). The resolution is shown on the y-axis for each of the images 600, 602, and 604. The input image 600 has a caliper measurement of 660 μm, and the fake CT image 604 has a caliper measurement of 626 μm ± 40 μm. As shown in the figure, the resolution of the NIR image is low (0-100 units), while the resolution of the pseudo-CT image is similar to that of CT image 602 (close to 300 units). The x-axis in images 600, 602, and 604 corresponds to the line dimensions of the layers of the product or part constructed by additive manufacturing.

[0033] Figure 6B shows Table 606 of measured values ​​for a surface (surface 1) of the product or part whose cross-section is shown in Figure 6A, according to several implementation configurations. Table 606 shows caliper values ​​in 26.14 units, with 1369 CT pixels, 428 NIR pixels, 1297 false CT pixels, a CT voxel size of 19.0 μm, an NIR voxel size of 61 μm, and a false CT voxel size of 20.15 μm. Table 606 shows the numbers used to calculate the voxel size of the image shown in Figure 6A, according to several implementation configurations.

[0034] Figure 7 shows exemplary thickness measurements 700 using NIR, CT, and sham CT data for the same locations as in the portions in Figures 6A and 6B, according to several implementation configurations. In Figure 7, according to several implementation configurations, graph 702 corresponds to the input NIR image 600, graph 704 corresponds to the CT image 602, and graph 706 corresponds to the sham CT image 604. Each graph or image contains maximum, minimum, and average thickness measurements across the region. The measurements were compared to caliper measurements from the same location to determine accuracy, as described above with reference to Figures 6A and 6B.

[0035] Figures 8A and 8B show examples of artificial high-resolution images according to several implementations. Figure 8A shows an NIR image 804, which is input to a trained network trained on CT data from an earlier construction to generate a pseudo-CT image 806 (an artificial image that matches the CT image). The resolution and / or quality of the pseudo-CT image will match that of the true CT image as the network converges. Figure 8B shows the output of the network trained as described with reference to Figure 8A, tested on different geometry (i.e., the network is tested on multiple geometrys not included in the training set). For example, using the trained network as described in Figure 8A, input image 808 generates image 810 and input image 812 generates image 814.

[0036] Figure 9 is a schematic diagram of system 900 for Z-axis enhancement (interpolation) according to several implementations. Some implementations use a separate neural network (e.g., a neural network other than GAN208 described above). Some implementations train and use GAN208 for Z-axis enhancement. In some implementations, a separate network is trained not only to improve resolution on the X and Y axes but also to interpolate between printed layers on the Z axis. Experiments have shown continuous GD&T accuracy in the results, and in some cases, pore definition is improved. In some implementations, the interpolation operation is performed by the interpolation module 344 described above with reference to Figure 3. In some implementations, three channel inputs (e.g., a 3x1 channel input including a first input channel 902 at 0 μm, a second input channel 904 at 25 μm, and a third input channel 906 at 50 μm) are combined to form a 3-channel input 908. In some implementations, one input channel is used for each layer of the additively manufactured product. In some implementations, a separate input channel is used for each layer at a predetermined distance from a predetermined location on the additively manufactured product. In some implementations, each input channel corresponds to a layer at a predetermined distance (e.g., 25 μm) from another layer of the additively manufactured product. The three-channel input is fed into a neural network (similar to GAN208) that is trained to interpolate between printed layers along the Z-axis (similar to creating a high-resolution image), and outputs a three-channel output 910 (image 912 at 0 μm, image 914 at 25 μm, and image 916 at 50 μm, respectively), which is split into a 3x1 channel output.

[0037] Figure 10 is a block diagram illustrating a system 1000 for training a generative adversarial network to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementations. In some implementations, the system 1000 (e.g., a computing device 300) implements a method for improving the image resolution of a temporal sequence of 2D images of additively manufactured products. The method includes the step of performing a series of steps for each of a plurality of additive manufacturing processes 1002. The series of steps includes (1016) obtaining a plurality of consecutive low-resolution 2D images 1004 (e.g., low-resolution 2D images 338 using near-infrared (NIR) images) of each product during each additive manufacturing process period. The series of steps also includes (1018) obtaining a corresponding high-resolution 3D image 1008 (e.g., high-resolution 3D image 340) of each product after the completion of each additive manufacturing process. The high-resolution 3D image includes a plurality of high-resolution 2D images corresponding to the low-resolution 2D images. The series of steps also includes the step (1006) of selecting one or more tile arrangement maps, which subdivide each of the low-resolution 2D images into multiple LR tiles and each of the corresponding high-resolution 2D images into multiple corresponding HR tiles. In Figure 10, the multiple LR tiles and multiple HR tiles are shown by tile 1010. In some implementations, the LR and HR tiles are stored separately. In some implementations, ordered pairs of LR and HR tiles are computed and stored. The series of steps also includes the step (1012) of constructing an image augmentation generator 1014 iteratively in a generative adversarial network using a training input containing the ordered pairs of corresponding LR and HR tiles (1020). In some implementations, the series of steps also includes the step (1022) of storing the image augmentation generator for later use to augment a sequence of low-resolution 2D images captured about a product during an additive manufacturing period.

[0038] In some implementations, a generative adversarial network (e.g., GAN208) includes a first neural network with an image augmentation generator (e.g., generator 212) and a second neural network with a discriminator (e.g., discriminator 216). In some implementations, the step of constructing the image augmentation generator includes repeatedly training the image augmentation generator to create candidate high-resolution 2D images (e.g., fake CT data) based on low-resolution 2D images, and training the discriminator to distinguish between the candidate high-resolution 2D images and 2D slices of the resulting high-resolution 3D images (true high-resolution images). Examples of the steps for training low-resolution and high-resolution 2D images are described above with reference to Figures 2A-2C, 6, 8A, and 8B, according to some implementations. In some implementations, the step of building an image augmentation generator ends when the output of the image augmentation generator is classified by the discriminator against true high-resolution 3D images for 50 percent of candidate high-resolution 2D images over several consecutive training iterations. For example, both models gradually become more accurate. Once the accuracy of both networks stabilizes, the models are considered to have "converged" or "fully trained". In some implementations, human developers accept the quality of "fake" data results.

[0039] In some implementations, each of a series of consecutive low-resolution 2D images 1004 is a near-infrared (NIR) image of each product during its respective additive manufacturing process period.

[0040] In some implementations, each of the high-resolution 3D images 1008 is generated based on performing a micro-CT scan of each product after the respective additive manufacturing process is completed (e.g., 20 μm resolution).

[0041] In some implementations, the method further includes a step of cropping a low-resolution 2D image and aligning it with a high-resolution 2D image before the step of subdividing into tiles. In some implementations, the method further includes a step of extending LR tiles and HR tiles to the training input by performing curvature deformation on some of the 2D images. In some implementations, one or more tile placement maps include multiple tile placement maps, each subdivided according to a different pattern. In some implementations, the step of selecting and subdividing a tile placement map (often called the tile placement step) is performed by the image processing module 322 (for example, using the image tile placement module 324). Examples of tiled images and tile placement operations are described above with reference to Figures 3 and 4 according to some implementations.

[0042] Figure 11 is a block diagram illustrating a system 1100 that uses a generator from a trained generative adversarial network (for example, a GAN trained via the process described above with reference to Figure 10) to improve the image resolution of a sequence of 2D images of additively manufactured products, according to several implementation forms. System 1100 implements a method provided for improving the image resolution of a sequence of 2D images of additively manufactured products. The method is performed on a computing device 300 having one or more processors and memory for storing one or more programs configured to run by the one or more processors.

[0043] The method includes the step (1118) of obtaining multiple consecutive low-resolution 2D images 1104 (e.g., near-infrared (NIR) images) of a product during a period of an ongoing additive manufacturing process 1102 (e.g., additive manufacturing process 336). The method also includes the step (1120) of obtaining a previously trained image augmentation generator 1014 (e.g., trained as described above with reference to Figure 10) as part of a generative adversarial network. The image augmentation generator is configured to accept input images of a fixed 2D size. The method also includes the step (1106) of selecting one or more tile arrangement maps (e.g., using an image tile arrangement module 324) that subdivide each of the low-resolution 2D images into multiple LR tiles 1108. Spatially corresponding tiles from the low-resolution 2D images form a stack of multiple tiles. The method also includes the step (1110) of applying the image augmentation generator 1014 to each of the LR tiles to generate high-resolution 2D artificial image tiles 1112 of the product. The method also includes the steps of stitching together tiles of the high-resolution 2D artificial image (for example, using the image stitching module 326) for a set of high-resolution 2D artificial layers corresponding to a low-resolution image, and stacking the high-resolution 2D artificial layers together to form a 3D artificial volume of the product.

[0044] In some implementations, the method further includes a step (1114) of using a 3D artificial volume to identify post-construction effects and / or defects 1116 in the product (e.g., detecting features such as cracks / pores and predicting post-layer effects such as remelting, expansion, or contraction based on previous construction).

[0045] In some implementations, a generative adversarial network (as described above with reference to Figure 2B) includes a first neural network with an image augmentation generator and a second neural network with a discriminator.

[0046] In some implementations, during a training period (an example of which is described above with reference to Figure 10), the image augmentation generator 1014 is trained to generate candidate high-resolution 2D images (e.g., spurious or artificial CT data) based on low-resolution 2D images (e.g., NIR images), and the discriminator is trained to discriminate between the candidate high-resolution 2D images and slices of true high-resolution 3D images captured after the additive manufacturing process is complete.

[0047] In some implementations, step 1104, which obtains multiple consecutive low-resolution 2D images, includes capturing a separate low-resolution 2D image (for example, the images shown in Figures 4, 5, 6, 8A, 8B, and 9) for each layer of the product during the duration of the additive manufacturing process.

[0048] In some implementations, the method further includes the step of resizing multiple consecutive low-resolution 2D images (for example, using the image transformation module 328).

[0049] In some implementations, each tile layout map subdivides each of the low-resolution 2D images into non-overlapping tiles. In some implementations, one or more tile layout maps include multiple tile layout maps, each subdividing the low-resolution 2D image according to a different pattern. In some implementations, the stitching step includes generating an output image for each pixel by calculating the sum of their respective weighted values ​​in the corresponding high-resolution 2D artificial image tile for each pixel that falls within two or more overlapping regions of the tile layout map. In some implementations, the step of calculating each weighted sum includes associating each tile's contribution to each weighted sum with a weight that is linearly proportional to the distance from the center of each tile.

[0050] In some implementations, the method further includes the step of converting the high-resolution 3D artificial volume of the product into a specific CT scan format (for example, a format usable by commercially available CT analysis tools) (for example, using the image conversion module 328).

[0051] In some implementations, the method further includes a step of interpolation between the printed layers in the ongoing additive manufacturing process using a trained neural network (for example, using interpolation module 344, as described above with reference to Figure 9) (for example, the GAN network also performs interpolation between printed layers on the Z axis in addition to improving resolution on the X and Y axes).

[0052] Several implementations repeat the process described with reference to Figure 11, then label the defective parts and / or remove them. For example, some systems identify and / or discard products that are expected to exceed a predetermined porosity threshold or layers that do not meet predetermined geometric criteria.

[0053] The terms used herein in describing the present invention are for the sole purpose of describing specific implementations and are not intended to limit the invention. The singular forms “a,” “an,” and “the” used herein and in the appended claims are intended to include the plural forms as well unless the context clearly indicates otherwise. The terms “and / or” as used herein will also be understood to refer to and encompass any one or more possible combinations of the related listed items. The terms “comprises” and / or “comprising,” as used herein, specify the presence of a particular feature, step, action, element, and / or component, but will not be understood to exclude the presence or addition of one or more other features, steps, actions, elements, components, and / or groups thereof.

[0054] The above description is written with reference to specific implementations. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the exact form disclosed. In light of the above teachings, many modifications and variations are possible. These implementations have been selected and described to best illustrate the principles of the invention and its practical applications, thereby enabling those skilled in the art to best utilize the invention and various modifications to suit their specific intended uses. [Explanation of Symbols]

[0055] 100 Systems 102 Additive manufacturing equipment 104 Camera Devices 106 Image Sensor 108 Image Processors 110 Communication Network 112 Field inspection server / engine, inline monitoring engine 114 Computer Vision Processing Algorithms 116 Machine / Deep Learning Data Models 118 Additive Manufacturing Parameters 200 Systems 202 Raw input images, raw NIR data, raw CT data 204 NIR / CT data 206 Intermediate tile placement input, xy / yz / xz tiles 208 Generative Adversarial Networks (GANs), GAN Generators 210 Input data, latent random variables 212 Generator 214 True Data Samples 216 Discriminant 218 samples, fake samples 220 blocks, conditions 222 Fine-tuning feedback 224 System 226 Raw input image, raw NIR image 228 NIR images 230 ordered tiles 232 Ordered tile output 234 Output 236 Digital Twin Model 300 computing devices, computer systems 302 Processing Units / Cores, CPU, GPU 304 Communication Interface, Communication Network Interface 306 User Interface 308 Display devices, displays 310 Input Devices / Mechanisms 312 Communications Bus 314 memory 316 Operating Systems 318 Communication Module 320 data visualization applications and modules 322 Image Processing Module 324 Image Tile Placement Module 326 Image Stitch Module 328 Image Conversion Module 330 Additive Manufacturing Control Module 332 Tile Placement Map 334 Machine Learning / Deep Learning / Regression Models 336 Additive Manufacturing Process 338 Low-resolution 2D images 340 high-resolution 3D images 342 Post-construction effect model 344 Interpolation Module 346 Input / Output User Interface Processing Module 400 images 402 Images 404 Image 406 images 500 stitching algorithms, images 600 images 602 images 604 images 600 Input NIR Images 602 CT images 604 Fake CT images 606 table 700 Thickness measurement 702 Graph 704 Graph 706 Graph 804 NIR image 806 Fake CT images 808 input images, NIR 810 images, fake CT 812 images 814 images 900 System 902 First input channel 904 Second input channel 906 Third input channel 908 3-channel input 910 3-channel output 912 images 914 images 916 images 1000 System 1002 Additive Manufacturing Process 1004 Low-resolution 2D images 1006 steps 1008 high-resolution 3D images 1010 tiles 1012 Image Augmentation Generator 1014 Image Augmentation Generator 1100 System 1102 Ongoing additive manufacturing processes 1104 Low-resolution 2D image 1106 Steps 1108 LR Tiles 1110 steps, image augmentation generator 1112 High-resolution 3D artificial image tiles, high-resolution 3D artificial images 1114 steps 1116 Post-construction effect, defect 1118 steps 1120 steps

Claims

1. A method for improving the image resolution of a sequence of 2D images of additively manufactured products, For each of the multiple additive manufacturing processes, During each additive manufacturing process, the process involves obtaining multiple consecutive low-resolution 2D images of each product, A step of obtaining a high-resolution 3D image of each of the products after the completion of each of the additive manufacturing processes, wherein the high-resolution 3D image includes a plurality of high-resolution 2D images corresponding to the low-resolution 2D image. A step of selecting one or more tile arrangement maps that subdivide each of the low-resolution 2D images into a plurality of LR tiles and each of the corresponding high-resolution 2D images into a plurality of corresponding HR tiles, wherein the one or more tile arrangement maps include a plurality of tile arrangement maps, each subdivided according to a different pattern. The steps include: constructing an image augmentation generator repeatedly in a generative adversarial network using a training input that includes ordered pairs of corresponding LR and HR tiles; The steps include storing the image augmentation generator for later use to augment a sequence of low-resolution 2D images of the product captured during the additive manufacturing period, and Methods that include...

2. The method according to claim 1, wherein each of the plurality of consecutive low-resolution 2D images is a near-infrared (NIR) image of the respective product captured during a time sequence of the respective additive manufacturing process.

3. The method according to claim 1, wherein each of the high-resolution 3D images is generated based on performing a micro-CT scan of each of the products after the completion of each of the additive manufacturing processes.

4. The method according to claim 1, wherein the generative adversarial network includes a first neural network comprising the image augmentation generator and a second neural network comprising a discriminator.

5. The step of repeatedly constructing the aforementioned image expansion generator is The steps include training the image augmentation generator to create candidate high-resolution 2D images based on low-resolution 2D images, The steps include training the discriminator to distinguish between the candidate high-resolution 2D image and the 2D slice of the obtained high-resolution 3D image, and The method according to claim 4, including the method described in claim 4.

6. The method according to claim 5, wherein the step of constructing the image augmentation generator is completed when the output of the image augmentation generator is classified by the discriminator as true high-resolution 3D images for 50 percent of the candidate high-resolution 2D images over a series of consecutive training iterations.

7. The method according to claim 1, further comprising the step of cropping the low-resolution 2D image and aligning it with the high-resolution 2D image before the step of subdividing it into tiles.

8. The method according to claim 1, further comprising the step of extending the LR tiles and HR tiles to the training input by performing a curvature deformation on some of the 2D images.

9. Before the step of subdividing into tiles, the steps of cutting out the low-resolution 2D image and aligning it with the high-resolution 2D image, The steps include extending the LR tiles and HR tiles to the training input by performing curvature deformation on some of the 2D images, and The method according to claim 1, further comprising:

10. A method for improving the image resolution of a sequence of 2D images of additively manufactured products, The steps include obtaining multiple temporally consecutive low-resolution 2D images of the product during the ongoing additive manufacturing process, The steps include obtaining a previously trained image augmentation generator as part of a generative adversarial network, wherein the image augmentation generator is configured to accept an input image of a fixed two-dimensional size, A step of selecting one or more tile arrangement maps that subdivide each of the low-resolution 2D images into a plurality of LR tiles, wherein the one or more tile arrangement maps include a plurality of tile arrangement maps, each subdividing the low-resolution 2D image according to a different pattern. To generate high-resolution 2D artificial image tiles of the said product, the steps include applying the image enhancement generator to each of the LR tiles, With respect to the set of high-resolution 2D artificial layers corresponding to the low-resolution 2D images, the steps include stitching together the high-resolution 2D artificial image tiles, The steps include stacking the high-resolution 2D artificial layers together to form a 3D artificial volume of the aforementioned product, and Methods that include...

11. The method according to claim 10, wherein the generative adversarial network includes a first neural network comprising the image augmentation generator and a second neural network comprising a discriminator.

12. During the training period, The aforementioned image augmentation generator is trained to generate candidate high-resolution 2D images based on low-resolution 2D images. The method according to claim 11, wherein the discriminator is trained to discriminate between a candidate high-resolution 2D image and a slice of a true high-resolution 3D image captured after the additive manufacturing process is completed.

13. The method according to claim 10, wherein the step of obtaining a plurality of temporally consecutive low-resolution 2D images includes the step of capturing a low-resolution 2D image for each layer of the product during the period of the ongoing additive manufacturing process.

14. The method according to claim 10, further comprising the step of changing the size of a plurality of temporally consecutive low-resolution 2D images.

15. The method according to claim 10, wherein each tile arrangement map subdivides each of the low-resolution 2D images into tiles that do not overlap.

16. The stitching steps are, For each pixel in two or more overlapping regions of the tile arrangement map, the sum of the respective weighted values ​​in the corresponding high-resolution 2D artificial image tile is calculated to generate the respective output image for each pixel. The method according to claim 10, including the method described in claim 10.

17. Calculating the weighted sum of each of the above is For each of the aforementioned weighted sums, the contribution of each tile is associated with a weight that is linearly proportional to the distance from the center of each tile. The method according to claim 16, including the method described in claim 16.

18. The method according to claim 10, further comprising the step of converting the 3D artificial volume of the product into a specific CT scan format.

19. Interpolation step between printed layers in the ongoing additive manufacturing process using a trained neural network The method according to claim 10, further comprising:

20. The method of claim 10, further comprising the step of using the 3D artificial volume to identify post-construction effects or defects in the product.

21. An electronic device for improving the image resolution of a sequence of 2D images of additively manufactured products, One or more processors, A memory that stores one or more programs configured to be executed by the one or more processors An electronic device comprising, wherein one or more programs include instructions for carrying out the method described in any one of claims 1 to 20.

22. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device, wherein the one or more programs include instructions for carrying out the method according to any one of claims 1 to 20.

Images