Monitoring and inspection of manufacturing processes based on co-registration of various sensor data

Abstract

A method of manufacturing a physical object includes capturing image data of at least a portion of the physical object and other sensor data related to at least a portion of the machine or the physical object during a process of manufacturing the physical object by a machine. The method further includes, during the process of manufacturing the physical object, co-registering the image data and the other sensor data on a pixel-by-pixel basis for each of a plurality of pixels of the image data, storing the co-registered image data and the other sensor data in a mutually associated state within a data structure, and using at least a portion of the co-registered image data and the other sensors to detect an anomaly within the physical object or within the process of manufacturing the physical object.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This patent document claims the benefit of priority of U.S. Patent Application No. 17 / 828,920, filed on May 31, 2022. The entire content of the application described above is incorporated by reference as part of the disclosure of this document.

[0002] (Statement Regarding Federally Sponsored Research) This invention was made with government support under Contract No. DE - AC52 - 07NA27344 awarded by the U.S. Department of Energy. The government has certain rights in this invention.

[0003] (Field of the Invention) At least one embodiment of the present invention relates to manufacturing techniques, and more specifically, to techniques for manufacturing monitoring and inspection of manufacturing processes based on co - registration of various sensor data.

Background Art

[0004] (Background) Manufacturing processes are inherently susceptible to variations, and as a result, the quality of manufactured parts or assemblies is also inherently prone to fluctuations. Thus, part inspection is routinely performed in manufacturing processes to assess whether the manufactured parts meet all specifications within tolerance. However, the specific inspection protocols can vary between processes and / or between parts.

[0005] For example, a high-throughput assembly line that produces the same version of the same part may require only a small but statistically significant / representative percentage of final product inspections to infer the part quality and production efficiency of the entire lot. Conversely, inspections may occur for each part in high-value boutique production, where part designs can vary across all processing steps, or in an additive manufacturing (AM) process. Also, quality inspections may occur throughout the build process and / or after part processing is complete. Regardless of the method implemented, inspections remain important to the manufacturing process from perspectives such as part appraisal, process quality evaluation, error, and root cause analysis, and are often the main obstacle within it.

[0006] In this context, many forms of inspection exist. Inspections can be characterized, for example, from the perspective of the sensing probes used, such as non-contact (confocal scanning, photography, hyperspectral imaging, X-ray scanning, etc.), contact (kinematic or electrical resistance probes), or both. Such sensing probes can collect the necessary measurements regarding parameters such as geometric dimensions and tolerances, mass, density, minimum thickness, coefficient of friction, color, chemical composition, electrical properties, odor, or any comprehensive set of specifications that must be met. Depending on the quality metric, an inspection process, such as the yield stress of a support beam, can be non-destructive or destructive. Digital cameras represent an omnipresent sensing modality and will likely remain so, especially due to the emergence of machine learning and computer vision-based classification and regression algorithms that can easily extract quality metrics from images or videos, i.e., sequences of images.

[0007] Various hardware and sensors may be required to adequately quality evaluate the parts being inspected. For each inspection task, there is often a complementary analysis that extracts signals from relevant sensors and ultimately compares measurements to some specifications. However, in conventional manufacturing inspection processes, data streams are handled separately, e.g., using images to identify surface defects, infrared (IR) probes to inspect temperature throughout construction, mechanical motion profiles to scrutinize tool paths, etc.

Summary of the Invention

Means for Solving the Problems

[0008] In at least some implementations, the techniques introduced here include a method of manufacturing a physical object. The method can include, during the process of manufacturing a physical object by a machine, a) image data of at least a portion of the physical object from an imaging sensor (e.g., a conventional visible spectrum digital camera), and b) simultaneously capturing other sensor data related to at least a portion of the machine or the physical object. "Other sensor data" can include data from one or more non-imaging sensors (e.g., pressure sensors, temperature sensors, stage position and motion, etc.) as well as data from one or more other imaging sensors (e.g., IR cameras). The method further includes, at each of a plurality of time points during the process of manufacturing the physical object and for each of a plurality of pixels of the image data, co-registering (spatially associating) the image data and the other sensor data on a pixel-by-pixel basis, storing the co-registered image data and the other sensor data in an associated state within a data structure, and using at least a portion of the co-registered image data and the other sensors to detect anomalies within the physical object or within the process of manufacturing the physical object. In at least some implementations, the method can further include triggering an action in response to the detection of an anomaly within the physical object or within the process of manufacturing the physical object.

[0009] Co-registering the image data and other sensor data can include associating the other sensor data with that pixel at each of a plurality of points in time and for each of a plurality of pixels of the image data. More specifically, co-registering can further include identifying, for each of the plurality of pixels of the image data, a particular pixel to be associated with a particular sensor value of the other sensor data, and identifying can be performed by: a) calculating, for each of a plurality of orthogonal coordinate axes, the number of pixels occupied by a physical object within the image data along the coordinate axis and determining the number of pixels per unit length along the coordinate axis; and b) identifying a particular pixel to be associated with a particular sensor value based on a set of reference position coordinates, the current position coordinates of a part of the machine, and the number of pixels per unit length along the coordinate axis.

[0010] At least a portion of the co-registered image data and other sensors can be used to detect anomalies within a physical object or within a process for manufacturing a physical object, and detecting can be performed by identifying, within the image data of the physical object, particular pixels indicative of an anomaly and verifying position coordinates associated with the anomaly based on the position coordinates of the particular pixels, the number of pixels per unit length along a first coordinate axis, and a first set of reference position coordinates.

[0011] One or more embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0012]

Figure 1

[0013]

Figure 2

[0014]

Figure 3

[0015]

Figure 4

[0016]

Figure 5

[0017]

Figure 6

[0018]

Figure 7

[0019]

Figure 8

[0020]

Figure 9A

[0021]

Figure 9B

[0022]

Figure 9C

Figure 9D

Figure 9E

[0023]

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0024] (Detailed Description) (Overview) As described above, in conventional manufacturing inspection processes, data streams are treated separately. As a result, conventional manufacturing inspection methods do not utilize the outstanding information from each individual data stream with respect to a given quality metric. What is introduced herein is, therefore, a technique for co - registering, i.e., “fusing”, data streams from various sensing modalities in any given manufacturing process. This technique systematically and intuitively combines per - layer images with other in - field sensor data. The result is a rich image - like tensor with information going beyond what is captured by a standard 2 - D image containing grayscale or color (e.g., RGB) pixels. This technique can add the readings from one or more sensors collected at each location within the image to all of the pixels within the image. Merging all of the images and image pixel data collected from the sensor suite results in a synergistic data structure that brings all of the benefits of spatial depiction, along with the (potentially, higher - frequency) information collected by the sensors.

[0025] The techniques introduced here can operate at any scale and can thus be implemented using a sensing suite that can (a) contain any number of sensors, where cameras among them can be considered sensors, (b) on each frame of a movie (which is just a temporal series of images), (c) using imaging setups that include multiple and / or different types of cameras, such as grayscale, color, multispectral, hyperspectral, etc., and / or (d) using sensed data that is collected on an as-needed basis regardless of when the image is taken.

[0026] Since the generated tensors have a better quality information density than standard images, practical insights for manufacturing and inspection processes can be more easily extracted via advanced analysis approaches such as computer vision, machine learning, process modeling, etc. For example, the combined image and sensor data can be used to generate alerts for manufacturing defects, trigger corrective actions, and / or produce rich multi-parameter graphics, tables, etc.

[0027] The techniques introduced here can be applied in many different fields and contexts. One exemplary use is, as further explained below, in particular, for part and process quality control for additive manufacturing (AM) (also called "3D printing") in direct ink writing (DIW) additive manufacturing (DIW-AM). However, it should be noted that the techniques introduced here are not limited to AM. Generally, the techniques introduced here can be used to provide improved inspection capabilities and routines, process monitoring, improved / accelerated quality detection for additive and conventional manufacturing process traceability, rapid closed-loop control, science and physics-guided machine learning, in-flight metrology, and in-situ data and process monitoring.

[0028] (Exemplary implementation) The techniques presented herein include a methodology for combining different classes of sensor data. As a result, in the techniques presented herein, sensor readings and images are registered (“co-registered”) within the same spatial and temporal coordinate system and assigned to each pixel illumination and / or color intensity value representing the sensor readings recorded at each pixel location. This approach is superior to simply different sensors that record individual signal pairs over time, such as digital thermometers, pH probes, humidity sensors, microphones, etc. In the techniques presented herein, these types and / or other types of sensors can be registered in the temporal coordinate system at the start or end of construction (manufacturing process), even if they are collected at different rates. Multiple images (from the same or different cameras) are then spatially co-registered using spatial references within each image, such as the edges of the part being manufactured, fiducial markers, etc., and they can be set up from each other's perspectives.

[0029] Co-registering the data from the images and other sensors involves implementing a routine that starts at a known point in time t known in the manufacturing process. t known can be selected at the discretion of the user or designer of the manufacturing process, but selecting to implement the following sequence at t known enables the resulting data tensor to be used throughout the process. All images and sensors are referenced to t known regardless of when the data is collected by any of the sensors. t known In addition, fiducial points are also used as described below.

[0030] Figure 1 illustrates an example of part 3 of a DIW-AM system where the techniques introduced here can be implemented. As described, the techniques introduced here are more generally applicable to any other type of system that essentially processes physical objects, where the applicability is not limited to either DIW-AM or AM. In Figure 1, part 1 of the part being processed sits on a build plate 2. The DIW-AM system includes at least one extruder 4 for extruding material to form the object to be processed. The system further includes a camera 5 and one or more other sensors 6 (e.g., temperature sensors, pressure sensors, etc.). The system also includes a computer system (not shown in Figure 1) that controls the extruder 4 and receives data from the camera 5 and other sensors 6.

[0031] Figure 2 is a block diagram of a DIW-AM system such as that shown in Figure 1, further showing a computer system 21. The computer system 21 may control each of one or more extruders 22 and receive feedback therefrom, such as (x, y, z) position data. Additionally, the computer system 21 also receives output signals from one or more sensors 23, which may include a conventional visible spectrum digital camera and one or more other sensors. At least some of the sensors 23 can be mounted on or adjacent to one or more of the extruders 22.

[0032] FIG. 3 is a block diagram showing computer system 21 in more detail. As shown, computer system 21 in at least some implementations includes an AM control module 31, an inspection / monitoring module 32, a co-registration module 33, a sensor data acquisition module 34, and a concentrated image data storage unit 35. The AM control module 31 controls the extruder 22. The control of the extruder 22 by the AM control module 31 can be affected by the output of the inspection / monitoring module 32. For example, if the inspection / monitoring module 32 detects an abnormality within the object being processed or within the processing process itself, this may send a signal to the AM control module 31 to stop the extruder 22 or modify the extrusion process. The inspection / monitoring module 32 accesses the co-registered image data and other sensor data in the concentrated image data storage unit 35. Alternatively, or in addition, the inspection / monitoring module 33 receives such data directly from the co-registration module 32 and scrutinizes the data for indications of abnormalities. In at least some implementations, the inspection / monitoring module 32 may employ machine learning and / or other types of artificial intelligence (AI) methods to detect such abnormalities. The specific methods used by the inspection / monitoring module 32 to detect abnormalities are not deeply involved in the techniques introduced here and thus need not be disclosed in this specification. The sensor data acquisition module 34 inputs the signals output by cameras and other sensors, buffers those signals to the extent necessary, and converts them into a format (e.g., by performing analog / digital conversion if the signals are not yet in digital format) that can be used by the co-registration module 33. The co-registration module 33 receives digital image data and other digital sensor data and co-registers them spatially and temporally using techniques that will be described further below. The co-registered data is stored in the concentrated image data storage unit 35.

[0033] Figure 4 illustrates a simple example of a part manufactured by DIW-AM to show a method that can be used to associate a spatial reference point with data from an image and other sensors. In Figure 4, a part 41 having a simple approximate square occupancy area in the x-y plane with known side lengths is 3D printed by use of software-controlled DIW-AM. In at least some implementations, the techniques introduced here are (1) the average number of pixels per length of the sides of the square in the x-y plane, PixelsPerLength, using all four sides, and (2) the geometric center point (x known , y known , z known ) of the square within the three orthogonal axes x, y, and z. Any one of a variety of conventional image processing algorithms can be used to identify these parameters. Based on the calculated PixelsPerLength, the technique calculates the dimensions of the part within the image based on a unit of length (i.e., micron, millimeter, meter, etc.) for any relative motion of the sensor, tool head, motion platform, etc., and / or the number of pixels. Equally, PixelsPerLength can be used to calculate the number of pixels over any given distance. Another implementation uses a build plate with fiducial points thereon and / or a set grid pattern to identify the (x known , y known , z known ) coordinates and PixelsPerLength within the image. Once these metrics are determined, it becomes a matter of whether the position of the camera should remain fixed relative to the build area or whether corrections can be implemented based on the motion of the camera relative to (x known , y known , z known ).

[0034] (x known , y known , z known ), t known, and from PixelsPerLength, any pixel within an image (or a portion of the image) can be assigned one or more sensor output values. For example, following the DIW-AM embodiment of FIG. 4, during which sensor readings S i (t + t known ) are collected, at a time after the construction process, assume that the extruder is located at a new location (x + x known , y + y known , z + z known ) relative to a known reference location. From this information, the pixel coordinates of the pixel to which S i (t + t known ) will be associated can be determined by Equation (1).

Number

[0035] Since this process acts reversibly, if a pixel (e.g., a pixel indicating an anomaly) is known within the image, its spatial coordinates (x + x known , y + y known ) can be determined by Equation (2).

Number

[0036] Equations (1) and (2) assume that the image is taken from a planar (or "bird's-eye") view, but can be easily adapted using any camera orientation, i.e., for or with respect to an elevation view of the side of the part (x known , y known , z known ), t known , and PixelsPerLength are determined for several images. In any case, the sensor output value S i (t + t known ) can then be associated with each pixel location in addition to the existing intensity and / or color values of the pixel.

[0037] Figure 5 conceptually illustrates an example of a method in which other sensor data can be stored in association with pixel (image) data. A tensor 51 can be used to store data. One or more dimensions 52 of the tensor 51 can be used to store the intensity and color values of individual pixels of a 2D image captured at time t + t known One or more dimensions 52 of the tensor 51 can be used to store the output value S of one or more sensors i (t + t known ). While the tensor 51 in FIG. 5 is illustrated as three-dimensional, it should be noted that in practice, the tensor used for this purpose can have any number of dimensions (e.g., to adapt to a manufacturing system with any number of sensors). Additional dimensions may be included to store, for example, a timestamp indicating when the 2D image and / or the image or other sensor output was captured (e.g., to identify relevant layers from among successive build layers in an AM process).

[0038] The following are some examples of other sensor-based information that can be obtained and co-registered using the images within the tensor in a manufacturing process, namely, temperature, humidity, accelerometer values, machine encoder information (detailing the position, velocity, acceleration, etc. of all motion platforms), local conductivity, vibration, fluid properties, microphone / audio, pressure, tilt, etc. Separate images, whose pixel locations can be referred to as (x known , y known , z known ), can be combined at any location where there is an overlap within their spatial coordinates. Hyperspectral, multispectral, and IR cameras can capture a wider and / or higher-resolution range of the electromagnetic spectrum than conventional visible-spectrum digital cameras, so information from these other types of cameras can be used to enhance the output of conventional (typically less expensive) cameras where a common location can be identified.

[0039] Sensors often have different data collection rates. In the case of probes that scan the surface of what they measure, the scan rate for each probe is likely to vary and each may have its own data collection frequency. However, since the pixels in an image have a known length, i.e., 1 / PixelsPerLength, and / or there is a known dwell time associated with each pixel, there are several methods for attributing data to each pixel and the optimal approach will be application- and sensor-dependent. One method would be to attach all non-imaging sensor output values to each pixel, resulting in the highest possible data density per pixel. Additionally, or alternatively, if multiple readings are taken from a given sensor at different pixels in a given image, additional artificial sensor output values can be generated for other pixels where the sensor output values are interpolated and not spatially aligned with the actual sensor readings. Additionally, or alternatively, various statistical metrics such as mean, mode, standard deviation, kurtosis, or any other notable feature that can be derived from the variance of the data collected across each pixel can be assigned to the pixel. Image-like data structures such as those described above (e.g., tensors) can be stored in a database for later retrieval and subsequent processing. Such processing can involve reformatting the composition of the pixels, retroactively incorporating the collected sensor data, and / or adding new information from offline measurements.

[0040] Figure 6 is a flowchart illustrating the overall process according to at least some implementations of the techniques introduced herein. Process 600 may be implemented at least partially in real time within and as part of process 500 (the "construction process") for machining a physical object. First, at step 601, process 600 captures image data of at least a portion of the physical object and other sensor data related to at least a portion of the machine or physical object. Next, at step 602, for each of a plurality of pixels of the image data, the process co-registers the image data and the other sensor data on a pixel-by-pixel basis. At step 603, the process stores the co-registered image data and other sensor data in a mutually associated state within a data structure such as a tensor. At step 604, the process uses at least a portion of the co-registered image data and other sensors to detect anomalies within the physical object or within the process of manufacturing the physical object.

[0041] Figures 7 and 8 illustrate in more detail a portion of process 600 of FIG. 6 according to at least some implementations. More specifically, FIG. 7 illustrates an example of a data acquisition and co-registration process 700. At least a portion of process 700, as shown, is included within construction process 500 and can be implemented in real time as part of construction process 500. The illustrated process is based on an object to be machined with a simple approximate square footprint in the x-y plane. It should be appreciated that the process can be readily modified to accommodate objects having other more complex shapes.

[0042] First, at step 701, process 700 captures an initial image of the portion to be machined. The initial image will be used to identify the reference coordinates and dimensions described above. Thus, at step 702, the process determines the reference coordinates (x known , y known , z knownDetermine these coordinates. These coordinates can be any known coordinates. For example, if the object has a simple square contour (as shown in FIG. 4), the coordinates can be, for example, the geometric center of the object in the x-y plane at a given z value such as z = 0. Next, in step 703, the process calculates PixelsPerLength, which is the average number of pixels per side of the object in the x-y plane. Next, in step 704, the timing variable t is set equal to t known at which the data acquisition and co-registration part of the process then begins.

[0043] The data acquisition and co-registration part of this process captures image data, captures other sensor data (i.e., data other than image data), and co-registers the image data and other sensor data. These steps can occur in parallel and can occur asynchronously with respect to each other. More specifically, in step 705, this process waits until it is time to acquire image data. The timing of image data capture can be based on any of various criteria such as a set time, a specific frequency, etc., based on the current instrument position, based on the current stage or layer of the construction process, etc. At an appropriate time t, the image data is then captured in step 708, in addition to its associated spatial (x, y, z) coordinates and time stamp (t value). The spatial coordinates may correspond to the position of an object or part of a processing instrument at the time t when the data is captured (e.g., the position of the extruder of an AM machine). Similar to and in parallel with steps 705 and 708, this process waits in step 706 until it is time to acquire other (non-image) sensor data. Timing criteria similar to those described in relation to step 705 can be used in step 706. At an appropriate time, the other sensor data is then captured in step 709, in addition to its associated spatial (x, y, z) coordinates and time stamp (t value). Note that a given system can include multiple imaging sensors and / or multiple non-imaging sensors, at least some of which can capture data asynchronously with respect to other sensors. Thus, the process flow can include separate wait / data capture branches such as steps 706 and 709 for each individual sensor or for a group of sensors.

[0044] In parallel with steps 705, 706, 708, and 709, the process also waits in step 707 until it is time to co-register image and other sensor data. At an appropriate time, the image data and other sensor data are co-registered and stored within a data structure such as a tensor in step 710. Co-registration can be done by assigning non-imaging sensor data to each of one or more pixel values and, in at least one implementation, in the manner described above in relation to equation (1). The timing of co-registration can be based on any of a variety of criteria such as those mentioned above in relation to steps 705, 706, 708, and 709. Other criteria that can be used to trigger co-registration can include the amount of newly acquired data that has not yet been co-registered, the amount of memory space available for buffering data that has not yet been co-registered, and the like. Following co-registration and storage, if the construction process has not yet been completed in step 711, the process loops back through steps 705, 706, and 707. Otherwise, the process ends.

[0045] Note that in other implementations, the order of the steps described above may vary. Additionally, other implementations may include additional steps and / or some of the steps described above may be omitted.

[0046] FIG. 8 shows an example of a monitoring / inspection process for detecting defects or other anomalies within a processed object and / or within a construction process. In the illustrated implementation, the monitoring / inspection process 800 occurs in real time as part of the construction process 500. However, in other implementations, this can be performed offline, i.e., in batch mode, based on archived data. At step 801, the monitoring / inspection process 800 waits until it is time to evaluate sensor data, which may include image data, non-image data, or both. The timing for evaluating the sensor data can be based on any of various criteria such as a set schedule, a specific frequency, etc., or based on the current position, stage, or layer of the processing process, etc. At an appropriate time, the process accesses, at step 802, an appropriate subset of the stored co-registered image data and other sensor data. The process then determines, at step 803, whether the accessed co-registered data indicates an anomaly. Any of various methods can be used to make this determination, the details of which are not deeply relevant to the techniques presented herein. If the obtained co-registered data indicates an anomaly, the process outputs an alert, triggers a corrective action, and / or generates or updates a report at step 804. Next, if the construction process has not yet been completed at step 805, the process loops back to step 801. Otherwise, the process ends.

[0047] Figures 9A - 9E show additional examples of types of output that can be generated based on co - registered image data and sensor data. In the examples of these figures, the object being manufactured comprises a number of layers, each of which consists of a plurality of parallel linear strands of material, and the strands of each layer are non - parallel with respect to the strands in other layers. Figure 9A shows an example in which a plurality of sensor data points 91 are overlaid on an enlarged image of an object (in the x - y plane) being manufactured by DIW - AM. Interpolated sensor data points can be added between the actual sensor data points to provide a correlation for each (x, y) coordinate on the image. Figure 9B shows an image of the object in which a plurality of diagonal strands 94 of a particular layer are visible and a color or shading representing a sensor production data metric is applied to each point within the strands 94 of that layer according to a defined color or shading map 95. In the example of Figure 9B, the data metric can be, for example, dispenser pressure, temperature, or any of various other sensor - based data metrics. Figures 9C, 9D, and 9E show additional images of the object 92 similar to that in Figure 9B in which other layers are emphasized.

[0048] Figure 10 is a high-level block diagram of a computer system in which at least a portion of the techniques disclosed herein may be implemented. The computer system 100 in FIG. 10 may represent the computer system 21 in FIGS. 2 and 3. The computer system 100 includes one or more processors 101, one or more memories 102, one or more input / output (I / O) devices 103, and one or more communication interfaces 104, all of which are interconnected with each other through an interconnect 105. The processor 101 controls the overall operation of the computer system 100, including controlling its constituent components. The processor 101 may be, or may include, one or more conventional microprocessors, programmable logic devices (PLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. The one or more memories 102 store data and executable instructions (e.g., software and / or firmware) that may include software and / or firmware for implementing the techniques introduced above. The one or more memories 102 may be, or may include, any of various forms of random access memory (RAM), read only memory (ROM), volatile memory, non-volatile memory, or any combination thereof. For example, the one or more memories 102 may be, or may include, dynamic RAM (DRAM), static RAM (SDRAM), flash memory, one or more disk-based hard drives, etc. The I / O devices 103 provide access to the computer system 100 by a human user and may be, or may include, for example, a display monitor, an audio speaker, a keyboard, a touch screen, a mouse, a microphone, a trackball, etc. The communication interface 104 enables the computer system 100 to communicate with one or more external devices (e.g., an AM processing machine and / or one or more remote computers) via a network connection and / or a point-to-point connection.The communication interface 104 may be, or may include, for example, a Wi-Fi adapter, a Bluetooth® adapter, an Ethernet® adapter, a Universal Serial Bus (USB) adapter, or the like. The interconnect 105 may be, or may include, one or more buses, bridges, or adapters such as, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a PCI Extended (PCI-X) bus, a USB, or the like.

[0049] Unless contrary to physical possibility, it is to be recalled that (i) the methods / steps described herein may be implemented in any sequence and / or in any combination, and (ii) the components of the individual embodiments may be combined in any manner.

[0050] The machine-implemented operations described above can be implemented by a programmable circuitry programmed / configured by software and / or firmware, or by a fully dedicated circuitry, or by a combination of such forms. Such dedicated circuitry (where applicable) can be in the form of, for example, one or more Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), System-on-Chip Systems (SOCs), and the like.

[0051] The software or firmware for implementing the techniques introduced herein may be stored on a machine-readable storage medium and may be executed by one or more general-purpose or special-purpose programmable microprocessors. "Machine-readable medium" includes any mechanism that can store information in a form accessible by a machine (the machine can be, for example, a computer, a network device, a mobile phone, a personal digital assistant (PDA), a manufacturing apparatus, any device with one or more processors, etc.) as the term is used herein. For example, machine-accessible media include recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

[0052] Any or all of the features and functions described above may be combined with each other, except insofar as this may be otherwise described above or insofar as any such embodiments may be incompatible by virtue of their functions or structures, as will be apparent to those skilled in the art. It should be borne in mind that, unless contrary to physical possibility, (i) the methods / steps described herein may be implemented in any sequence and / or in any combination, and (ii) the components of individual embodiments may be combined in any manner.

[0053] Although the subject matter has been described in terms of structural features and / or acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as examples for implementing the claims, and other equivalent features and acts are intended to be within the scope of these claims.

Claims

1. A method for manufacturing a physical object, the method comprising: During the process of manufacturing the physical object by a machine, Capturing image data of at least a part of the physical object and other sensor data related to the machine or the at least a part of the physical object; For each of a plurality of pixels of the image data, co - registering the image data and the other sensor data on a pixel - by - pixel basis; Storing the co - registered image data and other sensor data in a mutually associated state within a data structure; Using at least a part of the co - registered image data and other sensor data to detect an abnormality within the physical object or within the process of manufacturing the physical object. A method comprising the above.

2. The method according to claim 1, further comprising triggering an action in response to detecting an abnormality within the physical object or within the process of manufacturing the physical object.

3. The method according to claim 1, wherein co - registering the image data and the other sensor data includes associating the other sensor data with a pixel for each of the plurality of pixels of the image data.

4. The method according to claim 3, wherein co - registering the image data and the other sensor data includes associating the other sensor data with each pixel of the image data for each of a plurality of time points.

5. Co - registering the image data and the other sensor data includes identifying, by a first coordinate axis, a specific pixel to be associated with a specific sensor value of the other sensor data therealong; Calculating the number of pixels occupied by the physical object within the image data along the first coordinate axis. Determining the number of pixels per unit length along the first coordinate axis based on the number of pixels occupied by the physical object in the image data along the first coordinate axis; Identifying the specific pixel to be associated with the specific sensor value based on a first reference position coordinate, a first current position coordinate of a part of the machine, and the number of pixels per unit length along the first coordinate axis; The method according to claim 1, comprising: **Claim 6** The co-registration further includes performing the calculation and the determination with respect to a second coordinate axis orthogonal to the first coordinate axis, and identifying the specific pixel to be associated with the specific sensor value further includes based on a second reference position coordinate, a second current position coordinate of a part of the machine, and the number of pixels per unit length along the second coordinate axis. The method according to claim 5. **Claim 7** Using at least a part of the co-registered image data and other sensor data to detect an abnormality in the physical object or in the process of manufacturing the physical object, Identifying a specific pixel indicating the abnormality in the image data of the physical object; Confirming the position coordinates associated with the abnormality based on the position coordinates of the specific pixel, the number of pixels per unit length along the first coordinate axis, and the first reference position coordinate; The method according to claim 5, comprising: **Claim 8** The data structure comprises a tensor. The method according to claim 1. **Claim 9** The sensor data comprises image data from at least one non-imaging sensor. The method according to claim 1. **Claim 10** The sensor data comprises data from a plurality of non-imaging sensors. The method according to claim 1.

11. The method according to claim 1, wherein the process for manufacturing the physical object is an additive manufacturing (AM) process.

12. The method according to claim 11, wherein the process for manufacturing the physical object includes a direct ink writing (DIW) process.

13. A non-transitory machine-readable storage medium storing instructions, the execution of the instructions in a processing system causing the processing system to perform operations associated with a process of manufacturing a physical object by a machine, the operations including: during the process of manufacturing the physical object by the machine, capturing image data of at least a part of the physical object and other sensor data related to the machine or the at least a part of the physical object, the other sensor data including data from a non-imaging sensor; co-registering the image data and the other sensor data; and the co-registering includes identifying, for each of a plurality of pixels of the image data, a particular pixel to be associated with a particular sensor value of the other sensor data, the identifying includes: calculating, for each of a plurality of coordinate axes, the number of pixels occupied by the physical object in the image data along the coordinate axis and determining the number of pixels per unit length along the coordinate axis; identifying the particular pixel to be associated with the particular sensor value based on a set of reference position coordinates, current position coordinates of a part of the machine, and the number of pixels per unit length along the coordinate axis; and being performed by. A non-transitory machine-readable storage medium.

14. The operations further include: Storing the co-registered image data and other sensor data in associated states within a data structure; Using at least a portion of the co-registered image data and other sensor data to detect anomalies within the physical object or within the process of manufacturing the physical object; The non-transitory machine-readable storage medium according to claim 13, comprising:

15. The non-transitory machine-readable storage medium according to claim 14, wherein the operation further comprises triggering an action in response to detecting an anomaly within the physical object or within the process of manufacturing the physical object.

16. Using at least a portion of the co-registered image data and other sensor data to detect anomalies within the physical object or within the process of manufacturing the physical object includes: Identifying specific pixels within the image data of the physical object that indicate the anomaly; Based on the position coordinates of the specific pixels, the number of pixels per unit length along the first coordinate axis, and a first reference position coordinate, confirming the position coordinates associated with the anomaly. The non-transitory machine-readable storage medium according to claim 14, comprising:

17. The non-transitory machine-readable storage medium according to claim 13, wherein the operation further comprises storing the co-registered image data and other sensor data in associated states within a tensor.

18. A manufacturing system, comprising: A machine for manufacturing a physical object; A plurality of sensors, including an imaging sensor and a non-imaging sensor; A processing system configured to perform operations during the process of manufacturing the physical object; and The operations include: Obtaining at least partial image data of the physical object from the imaging sensor and non-image sensor data from the non-imaging sensor; For each of a plurality of pixels of the image data, co-registering the image data and the non-image sensor data on a pixel-by-pixel basis; Storing the co-registered image data and non-image sensor data in a mutually associated state within a data structure; Using at least a portion of the co-registered image data and non-image sensors to detect anomalies within the physical object or within the process of manufacturing the physical object A manufacturing system comprising.

19. The manufacturing system according to claim 18, further comprising triggering an action in response to detection of an anomaly within the physical object or within the process of manufacturing the physical object.

20. The co-registering includes identifying specific pixels to be associated with specific sensor values of the other sensor data, The identifying includes For each of a plurality of coordinate axes, calculating the number of pixels occupied by the physical object in the image data along the coordinate axis and determining the number of pixels per unit length along the coordinate axis; Based on a set of reference position coordinates, current position coordinates of a part of the machine, and the number of pixels per unit length along the coordinate axis, identifying the specific pixels to be associated with the specific sensor values The manufacturing system according to claim 18, performed by.

21. Using at least a portion of the co-registered image data and non-image sensors to detect anomalies within the physical object or within the process of manufacturing the physical object includes Identifying specific pixels indicative of the anomaly in the image data of the physical object confirming the position coordinates associated with the abnormality based on the position coordinates of the specific pixel, the number of pixels per unit length along the first coordinate axis, and the first reference position coordinates The manufacturing system according to claim 20, comprising:

22. The manufacturing system according to claim 18, wherein the data structure comprises a tensor.

23. The manufacturing system according to claim 18, wherein the machine is designed to perform additive manufacturing (AM).

24. The manufacturing system according to claim 23, wherein the machine is designed to perform direct ink writing (DIW) AM.

Images