Deep learning based image enhancement for additive manufacturing
By using generative adversarial networks (GANs) to improve the resolution of NIR images and combining computer vision and machine learning, the problem of lack of efficient quality control in incremental manufacturing is solved, achieving low-cost, accurate defect detection and post-construction effect prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BWXT ADVANCED TECHNOLOGIES LLC
- Filing Date
- 2021-12-01
- Publication Date
- 2026-06-19
Smart Images
Figure CN116888620B_ABST
Abstract
Description
Technical Field
[0001] The publicly disclosed implementations generally relate to incremental manufacturing, and more particularly to systems, methods, and user interfaces for deep learning-based image enhancement for incremental manufacturing. Background Technology
[0002] Due to the complexity of incrementally manufactured parts (e.g., parts manufactured using 3D printing), nondestructive quality control methods are very limited. The most widely used nondestructive testing method is micro-CT (computed tomography) scanning. While this process offers higher geometric accuracy, it is very expensive and time-consuming, and cannot be scaled up to large parts made of dense materials (e.g., the process is affected by shadow artifacts in dense materials). Some systems use digital twin 3D volumes based on near-infrared (NIR) images. NIR post-processing provides excellent definition of pores or cracks, but provides less accurate geometric dimensions and tolerances (GD&T) and does not predict post-construction effects. 3D volumes are also limited by image resolution. Furthermore, such systems only capture data during the manufacturing process, meaning that changes that occur later are not captured. For example, these systems do not capture the remelting of metal that alters the final geometry. Summary of the Invention
[0003] In addition to the problems described in the background section, there are other reasons for the need for an improved system and method for inspecting incremental manufacturing quality. For example, because existing technologies rely on post-production analysis of incrementally manufactured products, contextual information for appropriate root cause analysis is lacking. This disclosure describes a system and method that addresses the shortcomings of conventional methods and systems.
[0004] This disclosure describes a system and method that addresses some of the shortcomings of conventional methods and systems. This disclosure uses a deep neural network technique called Generative Adversarial Networks (GANs) to simultaneously increase the resolution of NIR images, model post-construction effects (such as remelting or shrinkage), and transform the data into a format usable by readily available CT analysis tools. The techniques described herein enable large 3-D volumes to be processed by neural networks. This disclosure describes two different process pipelines. The first pipeline is used for training and testing (an ongoing process that improves the quality of the algorithm), and the second pipeline is used for field deployment.
[0005] According to some implementations, this disclosure uses computer vision, machine learning, and / or statistical modeling to build digital models for on-site inspection of incremental manufacturing quality. The techniques described herein can be used to enhance lower-resolution NIR images to CT-like quality and resolution. Additionally, the output can be analyzed using readily available CT scanning software. For highly accurate integrated GD&T, the techniques require negligible cost per build. The techniques enhance feature detection of characteristics such as cracks or pores. The techniques can also be used to predict post-layer effects, such as remelting, expansion, and contraction, based on training from previous builds. Unlike CT, systems based on the techniques described herein do not produce scanning artifacts and help reduce noise in the output.
[0006] According to some implementations, the present invention uses one or more cameras as sensors to capture sequential images (e.g., still images or videos) during incremental manufacturing of a product. Utilizing computer vision and machine / deep learning techniques, the time-series images are processed as multidimensional data arrays to generate relevant analysis and / or predictive insights. This includes locating specific features (e.g., defects) and determining the extent of those features to assess quality.
[0007] In some implementations, trained computer vision and machine / deep learning algorithms are used to process images of the ongoing incremental manufacturing process to generate defect characterizations. In other implementations, computer vision and machine / deep learning algorithms are trained based on images of the ongoing incremental manufacturing process to determine product quality.
[0008] Depending on the implementation, a method is executed on a computing system. Typically, a computing system comprises a single computer or workstation, or multiple computers, each with one or more CPUs and / or GPUs and memory. Implemented machine learning modeling methods generally do not require computing clusters or supercomputers.
[0009] In some implementations, a computing system includes one or more computers. Each computer includes one or more processors and memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing any of the methods described herein.
[0010] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured for execution by a computing system having one or more computers, each having one or more processors and memory. The one or more programs include instructions for performing any of the methods described herein.
[0011] Therefore, methods and systems for facilitating on-site inspection in incremental manufacturing processes are disclosed. The discussions, examples, principles, components, structures, features, arrangements, and processes described herein can be applied to, adapted to, and embodied in adaptive manufacturing processes. Attached Figure Description
[0012] To better understand the disclosed systems and methods, as well as other systems and methods, reference should be made to the following detailed embodiments in conjunction with the accompanying drawings, in which the same reference numerals always refer to corresponding parts.
[0013] Figure 1 It is a block diagram based on some implemented systems for on-site inspection of incremental manufacturing processes.
[0014] Figure 2A It is a block diagram of a system based on a sequence of 2-D images of an incrementally manufactured product, which is used to train a generative adversarial network (GAN) to enhance the product's image resolution.
[0015] Figure 2B It is based on some implementation Figure 2A The diagram shows a block diagram of a Generative Adversarial Network (GAN).
[0016] Figure 2C It is a block diagram of a system based on some implementations of a sequence of 2-D images of products for augmentation using generative adversarial networks (GANs) to enhance incrementally manufactured products.
[0017] Figure 3 It is a block diagram based on some implemented computing devices.
[0018] Figure 4 This is an example of a tile representing an image of layers in an incremental manufacturing process.
[0019] Figure 5 It is a diagram that stitches together image tiles of the incremental manufacturing process based on some implementation.
[0020] Figure 6A It is a diagram based on some implementations of wall thickness measurements for NIR, micro-CT scans, and artificial high-resolution images.
[0021] Figure 6B A table showing exemplary measurements based on a geometric comparison test, according to some implementations, is shown.
[0022] Figure 7 An exemplary visualization of wall thickness measurements based on some implementations is shown.
[0023] Figure 8A and 8BExamples of artificially created high-resolution images based on some implementations are shown.
[0024] Figure 9 This is a schematic diagram based on some implemented methods for Z-axis upscaling (interpolation).
[0025] Figure 10 This is a block diagram illustrating a system of image resolution sequences of 2-D images of incrementally manufactured products, based on some implementation of a trained generative adversarial network.
[0026] Figure 11 This is a block diagram illustrating a system that uses generative adversarial networks to enhance the image resolution of a sequence of 2-D images of incrementally manufactured products, based on some implementations.
[0027] Referring now to an implementation, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without these specific details. Detailed Implementation
[0028] Figure 1 This is a block diagram of a system 100 for on-site inspection of an incremental manufacturing process. The incremental manufacturing equipment 102 is monitored by one or more camera devices 104, each device 104 including one or more image sensors 106 and one or more image processors 108. Data collected by the camera devices is transmitted to an on-site inspection server 112 using a communication network 110. The incremental manufacturing equipment 102 uses a set of incremental manufacturing parameters 118, which can be dynamically updated by the on-site inspection server 112. The manufacturing parameters can include detailed process flows defining the materials used and how the process is performed.
[0029] The on-site inspection server 112 uses some standard computer vision processing algorithms 114 and some machine / deep learning data models 116.
[0030] The process captures images on-site during incremental manufacturing operations and applies standard image processing techniques to emphasize features (e.g., Gaussian blur, edge detection of electrodes and weld pools, signal-to-noise filtering, and angle correction). Temporal cross-correlation is used to align image stacks or video frames with the geometry. In some implementations, this information is fed to one or more mounted robotic cameras for accurate image capture. By obtaining the temporal derivatives of the images, the system converts the temporal image trend into a still signal. The system trains a convolutional neural network on sequentially lagging batches of images using 3D convolutions (e.g., pixel location, intensity, and color / spectral banding). In some implementations, a machine / deep learning data model 116 outputs the probability of certain events (e.g., the type of defect or / or).
[0031] Figure 2AThis is a block diagram of a system 200 based on a sequence of image resolutions of 2-D images of incrementally manufactured products, used to train a generative adversarial network (GAN) to enhance the product. Raw input images 202 (e.g., NIR images or CT data) are obtained from on-site monitoring of the incremental manufacturing process. Some implementations acquire initial data (raw input images 202) by capturing NIR images at a predetermined resolution (e.g., 60 μm resolution) while the product is being built. Some implementations perform micro-CT scans of the printed part or product at a second predetermined resolution (e.g., 20 μm resolution). Typically, micro-CT scans are generated after the product has been fully constructed. In some implementations, each NIR image corresponds to an XY cross-section (e.g., layer) of the product. The input images are cropped and aligned to obtain NIR and CT data 204, which are then tiled using a warp transform (e.g., random xy, yz, or xz tiles) to obtain intermediate tile inputs 206. These tile stacks (each tile stack representing a portion of the overall image) are used to train the generative adversarial network 208. For example, each NIR image corresponds to an XY profile or layer of the product, and a stack of CT images comprises scans of the product layer by layer from bottom to top (i.e., first the bottom layer, followed by subsequent layers, all the way to the top layer). Aligning NIR and CT scan images involves matching images layer by layer, from the bottom layer to the top layer. Some implementations augment input data (data used to train a neural network) by modifying the input data to increase the amount of data available for training. Some implementations rotate, reposition, and / or scale the input data to obtain further data for training. In some instances, these operations do not actually provide additional data. For example, rotation does not provide additional data or information when the input data or image contains circles or hexagons. Some implementations augment data by warping it. In some instances, warping produces additional information (e.g., H-shape) for training a neural network. Warping is used to augment training datasets to increase the diversity of training data. Warping (sometimes called perspective warping) involves affine transformations (one or more) of the input data. Some implementations warp NIR images (or patches derived from NIR images) and CT images (or patches derived from CT images) to produce corresponding warped images (or patches) for training. The warping operation takes a 2D image and projects it into 3D space, making the image appear as if it were viewed at a certain angle. Because the final image is projected back into two dimensions, warping produces an image that appears stretched, twisted, bent, or otherwise deformed. Some implementations perform the warping operation on both CT and NIR input data in the same way, thereby creating a paired ground truth for geometry or features not visible in the original data.
[0032] Some implementations pair data sources by assembling NIR images into 3-D volumes, aligning NIR and micro-CT volumes, and / or upscaling NIR images to CT resolution (e.g., using basic interpolation). Some implementations extract training sets by randomly selecting patches from the aforementioned paired 3-D volumes, randomly manipulating the data to augment the dataset, and / or splitting the dataset into training, testing, and validation datasets. Some implementations then use the training set to train GAN 208.
[0033] Figure 2B It is based on some implementation Figure 2A The diagram shows a block diagram of a Generative Adversarial Network (GAN) 208. A Generative Adversarial Network, or GAN, is a high-level method for training deep convolutional neural networks (CNNs). Instead of using a single network, two separate networks (generator 212 and discriminator 216) are trained. Generator 212 is the network that will be used after training. Generator 212 takes input data 210 (e.g., a near-infrared or NIR image), processes the input data 210, and generates samples 218 (e.g., fake or artificial CT data). Result 218 is considered "fake" data because it is generated (not an actual CT scan). Discriminator 216 receives either the original "real" sample 214 or the "fake" generated sample 218 (or both) and attempts to determine whether the data is "real" or "fake." The two networks are trained simultaneously. The generator 212 is trained based on its ability to "deceive" the discriminator 216 into believing that its data is "real," while the discriminator 216 is trained based on its ability to distinguish between "real" and "fake" data. This causes both models or networks to become increasingly accurate. When the accuracy of both networks has stabilized (e.g., the accuracy remains constant over several iterations), the model is considered "converged" or "fully trained" (the state shown in block 220). In some cases, convergence occurs when the discriminator is correct approximately 50% of the time. Before the network converges, the state shown in block 220 provides fine-tuning feedback 222 to the generator 212 and the discriminator 216. In some implementations, human developers verify the quality of the "fake" data results. During the GAN process, the network is trained using a training dataset (as described above) and its accuracy is measured using an independent verification dataset (also as described above). For each input, the trained model produces statistics (e.g., accuracy and / or loss). In some implementations, once training has reached a point where developers are confident the output quality is good, a completely separate dataset is used to perform tests. This dataset is essentially a trial run of the production model, testing not only the neural network itself but also the entire image processing and assembly pipeline.
[0034] Figure 2CThis is a block diagram of a system 224 for enhancing the sequence of image resolutions of 2-D images of an incrementally manufactured product using a Generative Adversarial Network (GAN) 208, based on some implementations. Raw input images 226 (e.g., NIR images) are obtained from field monitoring of the incremental manufacturing process of the product. Some implementations capture layered images of the product during the incremental manufacturing process. Some implementations resize the images (e.g., increase the size by a factor of 3). The input images are cropped to obtain cropped NIR images 228. Some implementations segment the cropped image 228 into numbered tiles (e.g., as 256x256 pixel tiles), extracting tiles for every predetermined number of pixels (e.g., every 128 pixels). In some implementations, the tiles are overlapped (e.g., 4 tiles overlapped) to eliminate edge effects. Subsequently, the cropped NIR images are sorted and / or block-divided to obtain ordered tiles 230. Ordered patches 230 are input to a trained GAN generator 208 (which operates in inference mode) to obtain ordered patch output 232. The ordered patch output 232 is concatenated and blended to obtain output 234, which is used to reconstruct the digital twin model 236 (e.g., the concatenated image is appended to form a 3D volume). References are made below to some implementations. Figure 4 and 5 Describe the details of the patching and image stitching algorithms. In some implementations, each patch is fed into the GAN individually. The input image is sliced into patches, each patch is processed by the GAN, and then the output patches are reassembled into the final image. This is done for each image in the image stack (of patches), and then finally, the stack of output images is combined into a 3D volume.
[0035] Figure 3 This is a block diagram illustrating a computing device 300 according to some implementation. Various examples of the computing device 300 include servers, supercomputers, desktop computers, cloud servers, and other high-performance clusters (HPCs) of computing devices. The computing device 300 typically includes: one or more processing units / cores (CPU and / or GPU) 302 for executing modules, programs, and / or instructions stored in memory 314 and thereby performing processing operations; one or more network or other communication interfaces 304; memory 314; and one or more communication buses 312 for interconnecting these components. The communication bus 312 may include circuitry for interconnecting system components and controlling communication between system components.
[0036] The computing device 300 may include a user interface 306, which includes a display device 308 and one or more input devices or mechanisms 310. In some implementations, the input device / mechanism includes a keyboard. In some implementations, the input device / mechanism includes a "soft" keyboard that is displayed on the display device 308 as needed, allowing the user to "press" "keys" that appear on the display 308. In some implementations, the display 308 and the input device / mechanism 310 include a touchscreen display (also known as a touch-sensitive display).
[0037] In some implementations, memory 314 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices. In some implementations, memory 314 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 314 includes one or more storage devices remotely disposed relative to GPU(one or more) / CPU(one or more) 302. Memory 314, or alternatively, the non-volatile storage devices(one or more) within memory 314, includes non-transitory computer-readable storage media. In some implementations, memory 314 or the computer-readable storage media of memory 314 stores the following programs, modules, and data structures or subsets thereof:
[0038] • Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks;
[0039] • Communication module 318 is used to connect computing device 300 to other computers and devices via one or more communication network interfaces 304 (wired or wireless) and one or more communication networks (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.).
[0040] • Optional data visualization application or module 320 for displaying visualizations of incremental manufacturing defects for on-site inspection;
[0041] • Input / output user interface processing module 346 allows users to specify parameters or control variables;
[0042] • On-site inspection engine 112 (sometimes referred to as the online monitoring engine, as mentioned above) Figure 1 (as described). In some implementations, the inspection engine 112 includes an image processing module 322 and / or an incremental manufacturing control module 330, as described below for... Figure 10 As described below. In some implementations, the image processing module 322 includes an image segmentation module 324, an image stitching module 326, and / or an image conversion module 328, as described in more detail below;
[0043] • The image segmentation module 324 uses the image segmentation module 324 to subdivide the low-resolution 2-D image to obtain low-resolution (LR) patches and / or subdivide the high-resolution 2-D image to obtain high-resolution (HR) patches;
[0044] • Machine / deep learning / regression models 334 (e.g., Figure 1 Model 116 in the text (and as further described below) includes weights and input vectors;
[0045] • Incremental manufacturing process 336;
[0046] • 338 low-resolution 2-D images in sequence;
[0047] • High-resolution 3-D image 340, including slices of high-resolution 2-D image;
[0048] • Optionally, post-construction effect model 342; and / or
[0049] • Interpolation module 344.
[0050] Each of the above-identified executable modules, applications, or sets of procedures may be stored in one or more of the previously mentioned storage devices and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., instruction sets) do not need to be implemented as separate software programs, procedures, or modules, and therefore, in various implementations, various subsets of these modules may be combined or otherwise rearranged. In some implementations, memory 314 stores a subset of the above-identified modules and data structures. Additionally, memory 314 may store other modules or data structures not described above. Further descriptions are provided below based on some implementations. Figure 3 The operation and data structure properties of each module are shown in the diagram.
[0051] Although Figure 3 The computing device 300 is shown, but Figure 3 This is intended more as a functional description of the various possible features than as a structural diagram of the implementation described herein. In fact, and as those skilled in the art will recognize, the items shown individually can be combined, and some items can be separated.
[0052] In some implementations, although not shown, memory 314 also includes modules for training and executing the above-mentioned references. Figure 1 and 2A -2C describes the model. In particular, in some implementations, memory 314 also includes a random sampling module, a decoding framework, one or more convolutional neural networks, a statistical support package, and / or other image, signal, or associated data.
[0053] Figure 4 This is an illustration of exemplary tile representations of layers in an incremental manufacturing process according to some implementations. According to some implementations, images 400, 402, 404, and 406 may correspond to different layers of the same product (or different products built using the same or different incremental manufacturing processes). According to some implementations, each image is tiled according to a tile size that is a fraction of the image size (e.g., a quarter or an eighth of the image size). Some implementations use the same tile size for all images. Some implementations use different tile sizes for different products and / or different incremental manufacturing processes. Figure 4 In the example, images 400, 402, 404, and 406 correspond to tiles from separate groups of tiles from the same image. Some implementations combine these groups of tiles to create a set of tiles that overlap in each direction. For example, high-resolution images typically include 10 or more total tiles on each axis, for a total of 400 tiles (4*100). Due to tile offsets, there are tiles at the edges that have half the thickness in one or more directions. In some implementations, GANs require tiles to have a uniform size, so these tiles are not included in the computation (e.g., cropped portions are padded with zeros), or the cropped portions are mirrored from the non-cropped portions.
[0054] Figure 5 This is an illustration of tiled images of layers from an incremental manufacturing process implemented in some methods. Some implementations use image stitching algorithms that provide excellent blending of tiled images without loss of detail compared to traditional methods. Each pixel in the output image is generated by a weighted sum of the values of that pixel in each of four or more overlapping tiles. The weight of each tile is linearly proportional to its distance from the center of the tile to eliminate edge artifacts. According to some implementations, Figure 5 An initial segmented image 502 is shown, which can be processed by a stitching algorithm 500 to produce a stitched segmented image. Figure 5 In the image 500, overlapping tiles are shown, with edges from each overlapping tile appearing in the image. Some implementations generate image 500 by stitching together four overlapping tile configurations and then mixing them together by taking the average or mean of the four images (each image corresponding to an overlapping tile). Figure 5 A second image 502 generated using an improved blending method is also shown. In some implementations, before stitching each group of tiles, the tiles are multiplied by a "blended tile" generated by a mathematical formula, an example of which is shown below:
[0055]
[0056] In equation (1) shown above, x and y are the x and y coordinates (of the pixels in the image), and width and height are the width and height (dimensions) of the tile corresponding to the pixel. In some implementations, equation (1) produces a weighted image with a scale from 0 to 1 determined by the x and y distances to the center of the image. In some implementations, the images produced by these four configurations are summed together, so that the total weight of each pixel is 1.
[0057] Figure 6A and 6B The results of testing the output of a trained neural network (e.g., GAN 208) according to some implementations are shown. According to some implementations, the test results are based on caliper measurements of NIR, CT, and fake CT (artificial images created by the generator of GAN 208). According to some implementations, the test results come from a geometric comparison test, which includes: calibrating the pixel size by measuring the full width of the part pixel by pixel, measuring the thickness of the thin wall (the smallest possible feature) using calipers, and comparing the system's output measurements with the wall thickness measured for NIR and micro-CT scans. Figure 6A The diagram shows a cross-section of a part manufactured using incremental measurements in the width direction, based on some implementations. The measurements in the width direction are divided into the average pixel width of this region in each image to determine the voxel size. By using the entire width of the part, the measurement error due to + / -1 pixel edge sharpness uncertainty is minimized to a fraction + / -2 / (total number of pixels). Figure 6A In the diagram, the first image 600 is an NIR image (a low-resolution input image containing a minimum of 427 pixels, an average of 228 pixels, and a maximum of 431 pixels). The second image 602 corresponds to a CT scan image (sometimes referred to as a CT image, which is a higher-resolution image containing a minimum of 1367 pixels, a maximum of 1373 pixels, and an average of 1369 pixels). The third image 604 corresponds to the output generated by the system (a fake CT image containing a minimum of 1294 pixels, a maximum of 1301 pixels, and an average of 1297 pixels). The resolution is shown on the y-axis of each of the images 600, 602, and 604. The input image 600 has a caliper measurement of 660 μm, and the fake CT image 604 has a caliper measurement of 626 μm ± 40 μm. As shown, the NIR image has a low resolution (0-100 units), while the fake CT image has a resolution similar to that of the CT image 602 (close to 300 units). The x-axis in images 600, 602, and 604 corresponds to the linear dimensions of layers of a product or part manufactured incrementally.
[0058] Figure 6B The cross-section of the figure is shown according to some implementations. Figure 6A Table 606 shows the measured values of a face (“face 1”) of a product or part. Table 606 shows caliper values of 26.14 units, with 1369 CT pixels, 428 NIR pixels, 1297 fake CT pixels, a CT voxel size of 19.0 μm, an NIR voxel size of 61 μm, and a fake CT voxel size of 20.15 μm. According to some implementations, Table 606 shows the values used for calculating... Figure 6A The numbers representing the voxel sizes of the image shown.
[0059] Figure 7 This illustrates usage based on some implementations. Figure 6A and 6B An exemplary thickness measurement value of 700 for the same location of the part in NIR, CT, and fake CT data. Figure 7 In some implementations, curve 702 corresponds to the input NIR image 600, curve 704 corresponds to the CT image 602, and curve 706 corresponds to the spoofed CT image 604. Each curve or image includes the maximum, minimum, and average thickness measurements over that region. These measurements are compared with caliper measurements from the same location to determine accuracy, as referenced above. Figure 6A and 6B As stated above.
[0060] Figure 8A and 8B Examples of artificially created high-resolution images based on some implementations are shown. Figure 8A NIR image 804 is shown. NIR image 804 is fed into a trained network trained on earlier constructed CT data, which generates a fake CT image 806 (an artificial image that matches the real CT image). When the network converges, the resolution and / or quality of the fake CT image matches the resolution and / or quality of the real CT image. Figure 8B This illustrates how the network performs when tested on different geometries (i.e., testing the network on multiple geometries not included in the training set), as shown in the reference. Figure 8A The output of the trained network. For example, using... Figure 8A The network trained as described above generates image 810 from input image 808, and generates image 814 from input image 812.
[0061] Figure 9This is a schematic diagram of a system 900 for Z-axis lift (interpolation) based on several implementations. Some implementations use a separate neural network (e.g., a neural network other than the GAN 208 mentioned above). Some implementations train a GAN 208 and use it for Z-axis lift. In some implementations, the separate network is trained to not only increase resolution on the X and Y axes but also interpolate between printed layers on the Z axis. In experiments, results show consistent GD&T accuracy, and in some instances, improved aperture clarity is shown. In some implementations, the interpolation operation is performed by the above-mentioned reference. Figure 3 The described interpolation module 344 performs. In some implementations, three-channel inputs (e.g., 3x1-channel inputs, including a first input channel 902 at 0 μm, a second input channel 904 at 25 μm, and a third input channel 906 at 50 μm) are combined to form a 3-channel input 908. Some implementations use one input channel for each layer of the incrementally manufactured product. Some implementations use corresponding input channels for each layer at a predetermined distance relative to a predetermined position of the incrementally manufactured product. In some implementations, each input channel corresponds to a layer at a predetermined distance (e.g., 25 μm) relative to another layer of the incrementally manufactured product. The 3-channel input is fed into a neural network (similar to GAN 208), which is trained to interpolate between printed layers in the Z-axis (similar to generating a higher resolution image) and outputs a 3-channel output 910, which is split into 3x1-channel outputs (images 912, 914, and 916 at 0 μm, 25 μm, and 50 μm, respectively).
[0062] Figure 10This is a block diagram illustrating a system 1000 for enhancing the image resolution of a sequence of 2-D images of an incrementally manufactured product by training a generative adversarial network according to some implementation. In some implementations, system 1000 (e.g., computing device 300) performs a method for enhancing the image resolution of a time series of 2-D images of an incrementally manufactured product. The method includes performing a series of steps for each of a plurality of incremental manufacturing processes 1002. The series of steps includes obtaining (1016) a plurality of corresponding sequences of low-resolution 2-D images 1004 of the corresponding product (e.g., low-resolution 2-D images 338 using near-infrared (NIR) images) during the corresponding incremental manufacturing process. The series of steps also includes obtaining (1018) a corresponding high-resolution 3-D image 1008 of the corresponding product (e.g., high-resolution 3-D image 342) after the completion of the corresponding incremental manufacturing process. The high-resolution 3-D image includes a plurality of high-resolution 2-D images corresponding to the low-resolution 2-D images. The series of steps further includes: selecting (1006) one or more patch maps, which subdivide each low-resolution 2-D image into multiple LR patches and each corresponding high-resolution 2-D image into multiple corresponding HR patches. Figure 10 In this implementation, the plurality of LR tiles and the plurality of HR tiles are indicated by tile 1010. In some implementations, the LR and HR tiles are stored separately. Some implementations compute and store ordered pairs of LR and HR tiles. The series of steps further includes iteratively constructing (1012) an image enhancement generator 1014 in a generative adversarial network using (1020) training input comprising ordered pairs of corresponding LR and HR tiles. In some implementations, the series of steps further includes storing (1022) the image enhancement generator for subsequent use in enhancing sequences of low-resolution 2-D images captured for the product during incremental manufacturing.
[0063] In some implementations, the generative adversarial network (e.g., GAN 208) comprises a first neural network containing an image augmentation generator (e.g., generator 212) and a second neural network containing a discriminator (e.g., discriminator 216). In some implementations, iteratively building the image augmentation generator includes: training the image augmentation generator to generate candidate high-resolution 2-D images (e.g., fake CT data) based on low-resolution 2-D images; and training the discriminator to distinguish between 2-D slices of the candidate high-resolution 2-D images and the resulting high-resolution 3-D images (real high-resolution images). According to some implementations, the above refers to... Figure 2A-2CSections 6, 8A, and 8B describe examples of training (low-resolution 2-D images and high-resolution 2-D images). In some implementations, building the image augmentation generator stops when the discriminator classifies the output of the image augmentation generator as a true high-resolution 3-D image for 50% of the candidate high-resolution 2-D images during multiple consecutive training iterations. For example, both models become increasingly accurate. When the accuracy of both networks has stabilized, the model is considered "converged" or "fully trained." In some implementations, human developers accept the quality of results obtained using "fake" data.
[0064] In some implementations, each of the plurality of sequenced low-resolution 2-D images 1004 is a near-infrared (NIR) image of the corresponding product during the corresponding incremental manufacturing process.
[0065] In some implementations, each high-resolution 3-D image 1008 is generated based on a micro-CT scan (e.g., 20 μm resolution) of the product performed after the corresponding incremental manufacturing process is completed.
[0066] In some implementations, the method further includes cropping the low-resolution 2-D image and aligning it with the high-resolution 2-D image before subdividing it into tiles. In some implementations, the method further includes augmenting the LR and HR tiles in the training input by performing a warp transformation on some of the 2-D images. In some implementations, the one or more tile maps comprise multiple tile maps, each tile map being subdivided according to a different mode. In some implementations, the selection of tile maps and subdivision (sometimes referred to as tile division) is performed by image processing module 322 (e.g., using image tile division module 324). References are made above according to some implementations. Figure 3 and 4 Examples of segmented images and segmentation operations are described.
[0067] Figure 11 The illustration is based on the use of a self-trained generative adversarial network according to some implementations (e.g., via the above reference). Figure 10 A block diagram of system 1100 describes a process of training a GAN (Generative Adversarial Network) generator to enhance the image resolution of a sequence of 2-D images of an incrementally manufactured product. System 1100 executes a method provided for enhancing the image resolution of a sequence of 2-D images of an incrementally manufactured product. The method is executed in a computing device 300, which has: one or more processors; and a memory storing one or more programs configured for execution by said one or more processors.
[0068] The method includes: acquiring (1118) multiple sequences of low-resolution 2-D images 1104 (e.g., near-infrared (NIR) images) of the product during an ongoing incremental manufacturing process 1102 (e.g., incremental manufacturing process 336). The method also includes: acquiring (1120) previously trained data (e.g., as referenced above) as part of a generative adversarial network. Figure 10 The method further includes: (1106) selecting (1108) one or more tile maps (e.g., using image tile module 324) that subdivide each low-resolution 2-D image into multiple LR tiles 1108. Spatially corresponding tiles from the low-resolution 2-D images form multiple tile stacks. The method further includes: for each LR tile, applying (1110) image enhancement generator 1014 to generate a high-resolution 2-D artificial image tile 1112 of the product. The method further includes: stitching the high-resolution 2-D artificial image tiles together (e.g., using image stitching module 326) for a set of high-resolution 2-D artificial layers corresponding to the low-resolution images; and stacking the high-resolution 2-D artificial layers together to form a 3-D artificial volume of the product.
[0069] In some implementations, the method further includes: using 3-D artificial volumes to identify (1114) post-construction effects and / or defects in the product (e.g., feature detection, such as cracks / pores, predicting post-layer effects, such as remelting, expansion, and contraction, based on previous construction).
[0070] In some implementations, generative adversarial networks (as mentioned above) Figure 2B The above includes a first neural network containing an image enhancement generator and a second neural network containing a discriminator.
[0071] In some implementations, during training (see above) Figure 10 During the training (described in the example), the image enhancement generator 1014 is trained to generate candidate high-resolution 2-D images (e.g., fake or artificial CT data) based on low-resolution 2-D images (e.g., NIR images); and the discriminator is trained to distinguish between the candidate high-resolution 2-D images and slices of real high-resolution 3-D images captured after the incremental manufacturing process is completed.
[0072] In some implementations, obtaining the plurality of sequenced low-resolution 2-D images 1104 includes capturing a corresponding low-resolution 2-D image of each layer of the product during the ongoing incremental manufacturing process (e.g., Figure 4 , 5 (Images shown in 6, 8A, 8B and 9).
[0073] In some implementations, the method further includes: adjusting the size of the plurality of sequence low-resolution 2-D images (e.g., using image conversion module 328).
[0074] In some implementations, each tile map subdivides each low-resolution 2-D image into non-overlapping tiles. In some implementations, the one or more tile maps comprise multiple tile maps, each subdividing the low-resolution 2-D image according to a different pattern. In some implementations, the stitching includes: for each pixel included in two or more overlapping regions of the tile map, generating a corresponding output image for the corresponding pixel by calculating a corresponding weighted sum of values in the corresponding high-resolution 2-D artificial image tile. In some implementations, calculating the corresponding weighted sum includes: for each tile's contribution to the corresponding weighted sum, associating a corresponding weight linearly proportional to the distance to the center of the corresponding tile.
[0075] In some implementations, the method further includes converting the product’s high-resolution 3-D artificial volume (e.g., using image conversion module 328) into a raw CT scan format (e.g., a format that can be used by readily available CT analysis tools).
[0076] In some implementations, the method further includes: interpolating between printed layers in the ongoing incremental manufacturing process using a trained neural network (e.g., using interpolation module 344, and as referred above). Figure 9 (For example, in addition to increasing the resolution on the X and Y axes, the GAN network also interpolates between printed layers on the Z axis.)
[0077] Some implementation iteration references Figure 11 The described process involves subsequently marking and / or rejecting defective parts. For example, some systems identify and / or discard products predicted to have a porosity exceeding a predetermined threshold or layers that do not meet predetermined geometric standards.
[0078] The terminology used herein in the description of the invention is for the purpose of describing a particular implementation only and is not intended to limit the invention. As used in the description of the invention and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that, as used herein, the term “and / or” refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.
[0079] For purposes of explanation, the foregoing description has been described with reference to specific implementations. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The described implementations were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best use the invention and its various implementations with various modifications suitable for the contemplated particular uses.
Claims
1. A method for enhancing the image resolution of a sequence of 2-D images of an incrementally manufactured product, the method comprising: For each incremental manufacturing process in a series of incremental manufacturing processes: During the corresponding incremental manufacturing process, obtain multiple sequences of low-resolution 2-D images of the corresponding product; After the corresponding incremental manufacturing process is completed, a corresponding high-resolution 3-D image of the corresponding product is obtained, wherein the high-resolution 3-D image includes a plurality of high-resolution 2-D images corresponding to the low-resolution 2-D image; A neural network model is built using training inputs including the low-resolution 2D image and the corresponding high-resolution 2D image; The neural network model is stored for subsequent use in enhancing sequences of low-resolution 2-D images captured for the product during incremental manufacturing. Multiple block maps are selected, wherein each low-resolution 2-D image is subdivided into multiple LR blocks and each corresponding high-resolution 2-D image is subdivided into multiple corresponding HR blocks, wherein each block map is subdivided according to a different mode; Before subdividing the image into tiles, the low-resolution 2D image is cropped and aligned with the high-resolution 2D image; The LR and HR patches are augmented by performing warping transformations on some 2D images to generate warped training pairs. The neural network model is an image enhancement generator that is iteratively built in a generative adversarial network, and the training input includes ordered pairs of corresponding LR and HR patches, the ordered pairs including warped training pairs; and For new products during incremental manufacturing, the method includes: Obtain multiple sequences of low-resolution 2D images; The image enhancement generator is applied to generate high-resolution 2-D artificial image tiles; The high-resolution 2-D artificial image tiles are stitched together to form a high-resolution 2-D artificial layer, wherein the stitching includes: for each pixel included in two or more overlapping regions of the tiled image, generating a corresponding output value by calculating a weighted sum, wherein the weight contributed by each tile is linearly proportional to the distance to the center of the corresponding tile according to the following formula: where x and y are the x, y coordinates of a pixel of the image, and width and height are the width and height of the tile corresponding to the pixel; and A trained neural network is used to interpolate between printed layers in the incremental manufacturing process to form a 3-D artificial volume of the product with enhanced z-axis resolution.
2. The method of claim 1, wherein each of the plurality of sequence low-resolution 2-D images is a near-infrared (NIR) image of the corresponding product captured in chronological order during the corresponding incremental manufacturing process.
3. The method of claim 1, wherein each high-resolution 3-D image is generated based on performing a micro-CT scan of the corresponding product after the completion of the corresponding incremental manufacturing process.
4. The method of claim 1, wherein the generative adversarial network comprises a first neural network including the image enhancement generator and a second neural network including a discriminator.
5. The method of claim 4, wherein iteratively constructing the image enhancement generator comprises: The image enhancement generator is trained to generate candidate high-resolution 2D images based on low-resolution 2D images; and The discriminator is trained to distinguish between the candidate high-resolution 2D image and 2D slices of the obtained high-resolution 3D image.
6. The method of claim 5, wherein the construction of the image enhancement generator ceases when, during multiple consecutive training iterations, the output of the image enhancement generator is classified by the discriminator as a true high-resolution 3D image for 50% of the candidate high-resolution 2D images.
7. The method of claim 1, wherein the plurality of sequenced low-resolution 2-D images obtained for the new product during the incremental manufacturing process are time-series.
8. The method of claim 1, wherein the image enhancement generator is configured to accept an input image of a fixed 2D size.
9. The method of claim 1, wherein obtaining the plurality of sequence low-resolution 2-D images comprises capturing a corresponding low-resolution 2-D image of each layer of the product during an ongoing incremental manufacturing process.
10. The method of claim 1, further comprising: Adjust the size of the multiple sequences of low-resolution 2D images.
11. The method of claim 1, wherein each block map subdivides each low-resolution 2-D image into non-overlapping blocks.
12. The method of claim 1, further comprising: The 3-D artificial volume of the product is converted into the original CT scan format.
13. The method of claim 1, further comprising: The 3-D artificial volume is used to identify post-construction effects or defects in the product.
14. An electronic device for enhancing the image resolution of a sequence of 2-D images of a product manufactured incrementally, comprising: One or more processors; and The memory stores one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method as described in any one of claims 1-13.
15. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, said one or more programs including instructions for performing the method as claimed in any one of claims 1-13.