Method, apparatus and system for closed loop control of a manufacturing process - Patents.com

Abstract

Broadly speaking, embodiments of the present technique provide a method, apparatus, and system for automatically detecting and correcting errors in manufacturing parameters of a manufacturing process using closed-loop control. Advantageously, the present technique not only monitors the manufacturing parameters, but also provides instructions to enable any unacceptable variations in the manufacturing parameters during the manufacturing process to be corrected.

Description

[Technical field]

[0001] The present technique relates generally to automated error detection and correction during a manufacturing process. In particular, the present technique provides a method, apparatus, and system for automatically detecting and correcting errors during an additive manufacturing process. [Background technology]

[0002] Additive manufacturing (AM), often referred to as 3D printing, is a method of producing parts and devices through the successive layering of materials. Producing components using this technique allows for the assembly of complex shapes and structures that cannot be achieved using traditional manufacturing methods. With the technology used in numerous applications from health care and medical devices to aerospace and robotics, additive manufacturing offers great opportunities for designing and manufacturing complex devices. To date, the technology has enabled rapid prototyping and product development, and is now beginning to be used for end-use production parts. However, the great power that AM brings with its large design and parameter space also leaves it vulnerable to manufacturing errors. Thus, a single part often requires multiple iterations to achieve a successful print, wasting valuable materials, energy, and time. For each of these errors, an experienced human operator is required to assess the cause of the error and adjust the appropriate parameters. Thus, the automation of the manufacturing process not only has the potential to speed up the manufacturing process, but also reduces the number of personnel required for successful operation.

[0003] Moreover, manufacturing parameters vary between printers and can change significantly depending on the material chosen. New materials for AM continue to be developed, including cellular, nanocomposites, or cement in construction. Many of these materials are highly sensitive to printing conditions, but at the same time, are intended to be used in such non-ideal conditions. Such complex manufacturing conditions include printing in complex lattices, printing in less stable environments (e.g., off- or on-body), or in multi-material structures, providing more opportunities for error.

[0004] Accordingly, applicants have identified a need for improved techniques for automated closed-loop control of manufacturing processes. Summary of the Invention [Means for solving the problem]

[0005] In a first approach of the present technique, a computer-implemented method for closed-loop control of a manufacturing process is provided, the method comprising: receiving at least one image of the manufacturing process at predefined time intervals during the manufacturing process; processing the at least one image at each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process; determining whether the predicted value of the at least one manufacturing parameter is within a predefined range of values; and generating instructions for corrective action when the predicted value is outside the predefined range of values.

[0006] Advantageously, the present technique not only monitors manufacturing parameters, but also provides instructions that allow any unacceptable variations in the parameters to be corrected during the manufacturing process or after the current iteration of the manufacturing process has ended (before beginning a subsequent iteration). In contrast, many existing approaches only monitor parameters and do not automatically correct for variations.

[0007] Advantageously, a single trained ML model is used to process the images and predict the values ​​of the manufacturing parameters, compared to known methods that use separate models to analyze the images and predict the parameters (and therefore may be slower or require more computational resources to implement and may be more difficult to train).

[0008] Furthermore, the present technique monitors the manufacturing parameters in real time or near real time while the manufacturing process is taking place. This means that if the manufacturing parameters are acceptable, it is assumed that the manufacturing process will proceed correctly and therefore there is no need to pause the process to inspect the object being manufactured. Similarly, if one or more manufacturing parameters are unacceptable, corrective action can be taken in real time or near real time. For manufacturing parameters that cannot be corrected in real time or near real time, the present technique allows corrective action to be taken between iterations of the manufacturing process. This can therefore make the manufacturing process more time-efficient. It can also be more energy- and material-efficient, since for example the number of times defective objects are produced can be reduced. In contrast, existing approaches often require the manufacturing process to be paused so that the object being manufactured can be inspected, which can introduce significant delays into the manufacturing process.

[0009] The at least one manufacturing parameter is a parameter of the manufacturing process that may be changed or controlled. The at least one manufacturing parameter may depend on the manufacturing process being used. Broadly speaking, the manufacturing parameters may be parameters that may be controlled or corrected in real time or near real time, or parameters that may only be controlled or corrected between iterations of the manufacturing process. For example, the manufacturing parameters that may be corrected in real time may be printing parameters (e.g., flow rate, speed, etc.), and the manufacturing parameters that may be corrected between iterations may be tool path / slicing parameters.

[0010] For example, when the manufacturing process is a material extrusion process, the at least one parameter may be any one or more of: flow rate, transverse speed / feed rate, Z offset, hot end temperature, bed temperature, layer height, line width, infill density, wall thickness, and retract setting, It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0011] In another example, when the manufacturing process is a stereolithography (SLA) process, the at least one parameter may be any one or more of exposure time, lift speed, lift distance, light off delay, layer height, wall thickness, and infill density. These parameters are all types of parameters that can be corrected in real time or near real time. It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0012] In another example, when the manufacturing process is a laser powder bed fusion (LPBF) process such as selective laser sintering (SLS) or selective laser melting (SLM), the at least one parameter may be any one or more of laser power, scan speed, hatch distance, stripe width, stripe overlap, layer height, and laser spot size. It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0013] In another example, when the manufacturing process is a milling or turning process, which may use a mill or lathe, the at least one parameter may be any one or more of feed rate, spindle speed, cutting depth, cutting width, coolant, and cutter selection (e.g., number / depth of grooves). It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0014] In another example, when the manufacturing process is a laser cutting process, the at least one parameter may be any one or more of the following: laser power, feed / scan speed, and laser focal length / height, It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0015] In another example, when the manufacturing process is a plasma cutting process, the at least one parameter may be any one or more of arc current, arc voltage, cutting speed, and nozzle height, It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0016] The step of determining whether the predicted value of at least one manufacturing parameter is within a predefined range of values ​​may comprise using a wide range of values ​​set by a human expert. That is, a human expert familiar with the manufacturing process may know how far the manufacturing parameters can deviate without affecting the quality or integrity of the manufactured object. The human expert may also identify which particular manufacturing parameters are important in the progression of errors in a particular manufacturing process, as well as the values ​​of those manufacturing parameters that may cause the errors to progress. Thus, a predefined range of values ​​for each manufacturing parameter being monitored and controlled may be provided to the model for use during the training stage and / or during inference / runtime.

[0017] The step of generating instructions for corrective action may comprise generating instructions for adjusting a value of at least one manufacturing parameter. This may be useful when at least one manufacturing parameter deviates outside a predefined range of acceptable values, yet the manufacturing process is not yet adversely affected. For example, when the manufacturing process involves 3D printing an object, if the object is not adversely affected or damaged by the deviation of the at least one manufacturing parameter, it may be useful to correct / adjust the parameter and continue 3D printing the object. In this case, the method may comprise receiving a confirmation that the value of the at least one manufacturing parameter has been adjusted, and processing the at least one image received after the confirmation is received using the trained machine learning ML model.

[0018] Alternatively, the step of generating instructions for corrective action may comprise generating instructions to stop the current manufacturing process. This may be useful when deviations of at least one manufacturing parameter outside a predefined range of acceptable values ​​cause the manufacturing process to be adversely affected. This may also be useful when the manufacturing parameters that need to be corrected cannot be corrected in real time or near real time. For example, when the manufacturing process involves 3D printing an object, it may not be useful to continue 3D printing the object when the object is adversely affected or damaged by deviations of at least one manufacturing parameter. Instead, it may be efficient in terms of cost, time, energy, and materials to stop 3D printing the object. The manufacturing process may be restarted from the beginning. To avoid the same error occurring when the next iteration of the manufacturing process is started, the step of generating instructions for corrective action (when the predicted value of at least one manufacturing parameter is outside the predefined range of values) may be performed before the next iteration is started. Thus, the manufacturing parameters are corrected between iterations of the manufacturing process.

[0019] When the predicted value is outside a predefined range of values, the step of generating instructions for corrective actions may comprise using actions prescribed by a human expert. That is, the instructions may be generated based on rules of thumb provided by a human expert. This is advantageous because the model may be able to detect an error but may not know the best way to correct the error, whereas a human expert in the particular manufacturing process being controlled will know the best way to correct the error. A human operator of the manufacturing process is routinely required to assess the cause of the error, adjust the appropriate parameters, and restart the process. Thus, as described below, the model may be trained using images that have been labeled with manufacturing parameters and rules of thumb from the expert, which allows the model to generate instructions to correct the error.

[0020] The step of receiving at least one image at a predefined time interval may comprise receiving at least one image at a predefined time interval between 1 second and 10 seconds. In some cases, the at least one image may be received at a predefined time interval of less than 1 second. For example, an image sensor may be used to capture images at a rate of 30 frames per second (30 fps). It will be understood that these are examples and that the predefined time intervals and any suitable time intervals may be used.

[0021] It may be useful to receive at least one image at predefined time intervals throughout the manufacturing process such that the overall duration of the manufacturing process is monitored and controlled. However, often times, when an error (i.e., deviation of at least one manufacturing parameter) occurs at the beginning of the manufacturing process, it may have a negative impact on the rest of the process if the error is not corrected or is not correctable. Therefore, it may be useful to monitor the beginning of the manufacturing process (i.e., an initial part thereof) since if an error occurs at this stage, it may be more efficient in terms of cost, time, energy, and materials to correct the at least one parameter or to stop the process. Thus, receiving at least one image at predefined time intervals may comprise receiving at least one image at predefined time intervals during at least an initial part of the manufacturing process.

[0022] The method may further comprise sending an instruction to pause the manufacturing process and performing the processing, determining, and generating steps while the manufacturing process is paused, which may be useful so that the manufacturing process does not continue to use potentially unacceptable manufacturing parameters.

[0023] Processing the at least one image using the trained machine learning ML model may comprise processing the at least one image using a classification module of the trained ML model to predict a value of the at least one production parameter. In this case, the classification module may classify the value of the at least one production parameter using discrete classification bins. For example, the flow rate may be classified as "low", "good", or "high". Alternatively, processing the at least one image using the trained machine learning ML model may comprise using a regression module to predict the value of the at least one production parameter. In this case, a continuous prediction may be output for the production parameter. For example, the flow rate may be classified as "37%, " "102%, " or "274%".

[0024] In some cases, the method may be performed in real time to enable real-time control of the manufacturing process. In these cases, corrective actions are performed in real time or near real time with respect to the current iteration of the manufacturing process. This may be possible if the trained machine learning model is, for example, part of or near the equipment used to perform the manufacturing process.

[0025] Alternatively, the method may be performed after a current iteration of the manufacturing process has finished to allow control of subsequent iterations of the manufacturing process. In these cases, the corrective action is performed with respect to the subsequent iterations of the manufacturing process. This may be useful when the trained machine learning model is, for example, not part of or near the equipment used to perform the manufacturing process. For example, when the trained ML model is remote from the equipment (e.g., located on a remote or cloud server), the time to send images to the remote server and to send instructions back to the equipment for corrective action may be too long for the manufacturing process to be effectively controlled in effective or near real-time. Therefore, it may be more useful to use information received for one iteration of the manufacturing process to control another subsequent iteration. This may also be useful when the corrective action is to stop the current iteration of the manufacturing process. In another example, an error in at least one manufacturing parameter may intensify over time (e.g., during an iteration of the manufacturing process) and / or the parameter may not be correctable in real time. For example, errors such as cracking and warping deformations, where stresses in an object being manufactured build up over time, cannot be corrected in real time, where corrective action can only be taken on subsequent iterations of the manufacturing process.

[0026] As mentioned above, the control method may be suitable for various manufacturing processes. For example, the manufacturing process may be an extrusion-based 3D printing process. It will be understood that this is an exemplary and non-limiting manufacturing process. When the manufacturing process is an extrusion-based 3D printing process, the at least one manufacturing parameter may be any of the following: flow rate, lateral speed or feed rate, Z-axis offset, hot end temperature, bed temperature, layer height, line width, infill density, wall thickness, and retract setting. It will be understood that this is a non-exhaustive and non-limiting list of manufacturing parameters.

[0027] In a second approach of the present technique, an apparatus for performing a manufacturing process using closed-loop control is provided, the apparatus comprising at least one processor coupled to a memory and configured to receive at least one image of the manufacturing process at predefined time intervals during the manufacturing process, process the at least one image at each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process, determine whether the predicted value of the at least one manufacturing parameter is within a predefined range of values, and generate instructions for corrective action when the predicted value is outside the predefined range of values.

[0028] The apparatus may further comprise at least one image capture device for capturing at least one image of the manufacturing process at predetermined time intervals. The image capture device may be any one of a camera, a light sensor, and an infrared sensor or an infrared camera.

[0029] The apparatus may be any one of an extrusion-based 3D printer, an additive manufacturing apparatus, a material extrusion apparatus, a stereolithography apparatus, a laser powder bed fusion apparatus, a milling apparatus, a turning apparatus, a lathe, a laser cutter, and a plasma cutter, it being understood that this is a non-exhaustive list of possible apparatus.

[0030] In a third approach of the present technique, a system for closed-loop control of a manufacturing process is provided, the system comprising: an apparatus for performing the manufacturing process, the apparatus comprising at least one image capture device for capturing at least one image of the manufacturing process at predefined time intervals and a communication module for transmitting the at least one image for processing; and a remote server coupled to a memory and comprising at least one processor configured to receive at least one image of the manufacturing process from the apparatus, process the at least one image at each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process, determine whether the predicted value of the at least one manufacturing parameter is within a predefined range of values, and generate instructions for corrective action when the predicted value is outside the predefined range of values.

[0031] The at least one processor may be further configured to transmit the generated instructions to a device.

[0032] In some cases, the steps performed by the at least one processor may be performed in real time and the generated instructions may be transmitted while the manufacturing process is ongoing.

[0033] The step of generating instructions for corrective action may comprise generating instructions for adjusting a value of at least one manufacturing parameter, in which case the at least one processor may be further configured to receive a confirmation from the device that the value of the at least one manufacturing parameter has been adjusted, and to process the at least one image received after the confirmation is received using the trained machine learning ML model.

[0034] Alternatively, the step of generating instructions for corrective action may comprise generating instructions to stop the current manufacturing process.

[0035] The steps performed by the at least one processor may be performed after a current iteration of the manufacturing process has finished and the generated instructions may be transmitted before a subsequent iteration of the manufacturing process begins, in which case the step of generating instructions for corrective action may comprise generating instructions for adjusting a value of at least one manufacturing parameter of the subsequent iteration of the manufacturing process.

[0036] In a fourth approach of the present technique, a computer-implemented method is provided for training a machine learning ML model to enable closed-loop control of a manufacturing process, the method comprising obtaining a training dataset comprising a plurality of images of the manufacturing process, each image being labeled with a plurality of manufacturing parameters associated with the manufacturing process and a timestamp, and training the machine learning ML model by inputting images from the training dataset into the ML model, processing the input images using a module of the ML model to identify one of the manufacturing parameters, predicting values ​​of each manufacturing parameter for the input image using a module of the ML model, comparing the predicted values ​​with a label of the image, and updating the ML model to reduce a difference between the predicted values ​​and the label of the image.

[0037] In one example, the ML model may comprise an attention module / layer and an attention mask, a convolutional layer, a dense connection layer, and a skip connection. In this case, training the ML model may comprise inputting an image from a training dataset into the ML model (where the image may be an individual image or frame, or a video comprising multiple frames), processing the input image using the ML model to identify one of the production parameters, predicting a value of each production parameter for the input image using the ML model, comparing the predicted value with the label of the image to generate a loss function, and using backpropagation to train the ML model to reduce the loss function. The processing step may comprise using an attention layer and an attention mask, a convolutional layer, a skip connection, and / or a dense connection layer. Similarly, the predicting step may comprise using any or all of the layers. It will be understood that this is just one exemplary architecture of the ML model, and other suitable architectures may be used.

[0038] The ML model may also be trained to generate corrective actions to correct errors at inference time, i.e., at inference time, the ML model may not only identify that an error has occurred (i.e., that a manufacturing parameter is outside a predefined range of acceptable values), but may also generate instructions to correct the error, as described above. Thus, the ML model may be trained using expert heuristics that indicate how the error may be corrected.

[0039] Transfer learning may be used to improve the accuracy of an ML model in detecting and correcting errors in a single part or a family of similar parts. This may comprise using a pre-trained network to generate the ML model and training the pre-trained network on data derived from manufacturing just that one part or family of parts. Thus, the training data used to train the pre-trained network may comprise images of a broad range of parts or objects and / or a narrow range of parts (e.g., one part or a family of related / similar parts).

[0040] In a related approach to the present technique, a non-transitory data carrier is provided that carries processor control code for implementing any of the methods, processes and techniques described herein.

[0041] As will be appreciated by one skilled in the art, the present technique may be embodied as a system, method, or computer program product. Accordingly, the present technique may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0042] Furthermore, the present techniques may take the form of a computer program product embodied in a computer-readable medium having computer-readable program code embodied therein. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above.

[0043] Computer program code for carrying out operations of the present technique may be written in any combination of one or more programming languages, including object-oriented and conventional procedural programming languages. Code components may be embodied as procedures, methods, etc., and may comprise subcomponents that may take the form of instructions or sequences of instructions, at any of the levels of abstraction ranging from direct machine instructions of a native instruction set to higher level compiled or interpreted language constructs.

[0044] Embodiments of the present technique also provide a non-transitory data carrier carrying code that, when executed on a processor, causes the processor to perform any of the methods described herein.

[0045] The technique further provides processor control code for implementing the above-mentioned methods, for example on a general-purpose computer system or a digital signal processor (DSP). The technique also provides a carrier, which when in operation, carries processor control code for implementing any of the above-mentioned methods, particularly on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, a CD-ROM or DVD-ROM, a programmable or read-only memory (firmware) such as a non-volatile memory (e.g., flash), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) for implementing embodiments of the technique described herein may comprise source, object, or executable code in a conventional programming language (interpreted or compiled) such as C or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very High Speed ​​Integrated Circuit Hardware Description Language). As one skilled in the art will appreciate, such code and / or data may be distributed among multiple coupled components in communication with each other. The technique may include a controller including a microprocessor, working memory, and program memory coupled to one or more of the system's components.

[0046] It will also be apparent to those skilled in the art that all or part of the logic methods according to the embodiments of the present technique may be suitably embodied in a logic device comprising logic elements for performing the steps of the above-described methods, and that such logic elements may comprise components such as logic gates, for example, in a programmable logic array or an application specific integrated circuit. Such logic configurations may further be embodied with elements empowered to temporarily or persistently establish logic structures in such arrays or circuits, for example, using a virtual hardware description language that may be stored and transmitted using a fixed or transmittable carrier medium.

[0047] In one embodiment, the technique may be implemented using multiple processors or control circuits, and may be adapted to operate on or integrated into the device's operating system.

[0048] In one embodiment, the technique may be realised in the form of a data carrier having functional data thereon, said functional data comprising a functional computer data structure for enabling said computer system to carry out all the steps of the above-mentioned method when loaded into and operated by a computer system or network.

[0049] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0050] [Figure 1] 1 is a flow chart of exemplary steps for closed-loop control of a manufacturing process. [Diagram 2] FIG. 1 is a block diagram of a system for closed-loop control of a manufacturing process. [Figure 3A] FIG. 1 shows an overview of the CAXTON system used for automated data collection. [Figure 3B]FIG. 1 shows an overview of the CAXTON system used for automated data collection. [Figure 3C] FIG. 1 shows an overview of the CAXTON system used for automated data collection. [Figure 3D] FIG. 1 shows an overview of the CAXTON system used for automated data collection. [Figure 3E] FIG. 1 shows an overview of the CAXTON system used for automated data collection. [Figure 3F] FIG. 1 shows an overview of the CAXTON system used for automated data collection. [Figure 4A] FIG. 1 illustrates an example architecture of a neural network used to perform closed-loop control. [Figure 4B] FIG. 13 shows the confusion matrix of the final network for each parameter. [Figure 5A] FIG. 1 illustrates the three stages of training a residual attention CNN using the CAXTON 3D printing parameters dataset. [Figure 5B] FIG. 1 illustrates the three stages of training a residual attention CNN using the CAXTON 3D printing parameters dataset. [Figure 5C] FIG. 1 illustrates the three stages of training a residual attention CNN using the CAXTON 3D printing parameters dataset. [Figure 5D] FIG. 1 illustrates the three stages of training a residual attention CNN using the CAXTON 3D printing parameters dataset. [Figure 6A] FIG. 1 illustrates a machine vision control system pipeline and feedback parameters. [Figure 6B] FIG. 1 illustrates a machine vision control system pipeline and feedback parameters. [Figure 6C]FIG. 1 illustrates a machine vision control system pipeline and feedback parameters. [Figure 7A] FIG. 1 illustrates printer and feedstock agnostic online parameter correction and discovery. [Figure 7B] FIG. 1 illustrates printer and feedstock agnostic online parameter correction and discovery. [Figure 7C] FIG. 1 illustrates printer and feedstock agnostic online parameter correction and discovery. [Figure 7D] FIG. 1 illustrates printer and feedstock agnostic online parameter correction and discovery. [Figure 7E] FIG. 1 illustrates printer and feedstock agnostic online parameter correction and discovery. [Figure 7F] FIG. 1 illustrates printer and feedstock agnostic online parameter correction and discovery. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] Broadly speaking, embodiments of the present technique provide a method, apparatus, and system for automatically detecting and correcting errors in manufacturing parameters of a manufacturing process using closed-loop control. Advantageously, the present technique not only monitors the manufacturing parameters, but also provides instructions to enable any unacceptable variations in the manufacturing parameters during the manufacturing process to be corrected.

[0052] Errors are a frequent occurrence in additive manufacturing AM processes, and major challenges regarding reliability and consistency remain to be resolved. Currently, extrusion printing processes are open-loop, and today's machines are not aware of the current printing state. Due to the frequency of errors in the manufacturing process, this is a significant limitation. Thus, a single part often requires multiple iterations to achieve a successful print, wasting valuable materials, energy, and time. For each of these errors, an experienced human operator is required to assess the cause of the error and adjust the appropriate parameters.

[0053] Distortion is one of the most common error modes, especially in the higher performance and higher temperature materials (e.g., PEEK, ULTEM) used in more costly manufacturing settings. Distortion deformation is caused by the shrinkage of the extruded filament, which occurs because the deposition process involves large temperature gradients that develop residual thermal stresses. Errors caused by the accumulation of internal stresses in the printed part take time to appear, and therefore it is difficult to quickly detect errors and determine their causes. Multiple factors, such as model size, number of layers, stack section length, bed and chamber temperature, and material linear shrink rate, affect the scale of distortion in the print.

[0054] Many different techniques have been developed to detect errors during or after printing. A wide variety of sensors have been used to monitor the process, e.g., acoustic, inertial, and current sensors. However, vision-based sensing techniques provide much richer information and can identify a wider selection of error types. This is especially the case for long-term errors, such as distortion, where the appearance of the error is offset from the time of material deposition. There is contemporary work that uses both traditional computer vision and deep learning techniques to monitor the printing process. The latter has the advantage of being more generalizable and robust to changing conditions compared to hand-crafted feature detectors. Moreover, over the past decade, various neural network architectures have revolutionized the field of pattern recognition. Deep convolutional neural networks have led to numerous breakthroughs in image classification and object detection. In particular, object detection networks offer enormous opportunities in automated monitoring of manufacturing, as they are trained to detect and locate instances of features in images.

[0055] Deep learning techniques are particularly interesting because their potential is much more generalizable to new materials and printers than handcrafted features. Such models are beginning to be applied to process monitoring in extrusion printers to enable real-time corrections, demonstrating that deep learning methods can be effective in in-situ monitoring. For these systems to be deployed in a production environment, they must work on a wide range of printers, with various camera positions and lighting conditions, in addition to working for any 3D shape printed from materials of different colors and properties. Finally, automated error detection and correction methods need to be scalable to allow easy deployment and to collect more data to further improve the deep learning models.

[0056] The present technique provides a low-cost and scalable method for extending any manufacturing process, such as thermoplastic extrusion 3D printing, with conventional object detection models capable of detecting distortion, a frequent error in filament-based AM. The development of the present method also results in the curation of the first large-scale labeled dataset of distortion examples for a wide variety of part geometries. Using this dataset, a single-stage deep convolutional neural network is trained to both detect and localize distortion features in first-sight images and provide a confidence level for its predictions. Unlike existing approaches for other error modalities, the approach presented here extracts additional data from the images to make an estimate regarding the severity of the distortion error present. This is achieved through the development of a set of statistically validated metrics capable of determining distortion severity both during printing and upon completion of the print.

[0057] The present technique provides an easily deployable method to augment manufacturing processes with convolutional neural networks (CNNs) to create self-learning robotic printers capable of online error detection and correction in addition to parameter discovery for new, novel manufacturing materials. This is achieved through the development of a system called CAXTON, or Collaborative Autonomous Extrusion Network. CAXTON is a fully autonomous system for connecting and controlling learning 3D printers that enables fleet data collection and collaborative end-to-end learning. Each printer in the network can continuously print and collect data due to a novel part removal system. In particular, CAXTON uses low-cost cameras, deep learning algorithms, and automated sample removal equipment to autonomously learn how to accurately identify and correct errors at low computational cost. Unlike existing deep learning AM monitoring efforts, which often use artificial labeling of errors to train algorithms, CAXTON labels errors for deviations from optimal printing parameters. Thus, uniquely, CAXTON not only knows how to identify various errors, but also how to correct them, because it knows how far the printing parameters are from their optimum by looking at the images. This classification method also allows for autonomous generation of training data, allowing for larger and more diverse data sets for better accuracy, as well as generalization to previously unseen manufacturing devices, camera positions, and materials. This work also advances the prior art by having a system capable of devising multiple solutions to resolve the same error, the first work capable of correcting multiple parameters simultaneously, and self-learning the interactions between various parameters. With this capability, CAXTON can discover parameter combinations for unseen manufacturing materials using different manufacturing paradigms.Finally, visualization methods are employed to gain insight into how the trained neural network performs; this transparency is essential for real-world end-use applications, especially in areas such as medical device production.

[0058] Data has been collected using CAXTON to compile the first large-scale, optical, in-situ process monitoring dataset, including over one million sample images and their respective labeled printing parameters from 192 prints of different geometries. This dataset allows training of deep residual attention models capable of detecting suboptimal printing parameters. Using these trained models, simultaneous online correction of multiple printing parameters for known thermoplastic feed materials or manufacturing materials is demonstrated. This control loop removes time-consuming constraints and reduces the occurrence of errors, improving the efficiency of the 3D printing process. Furthermore, it is demonstrated that the system can self-learn parameter combinations to autonomously print unseen feed materials with dramatically different properties in unknown setups. This takes the place of a human expert operator and unlocks the widespread possibility of printing without human interaction. In addition to figuring out the best way to manufacture devices using optimization, it is hoped that this approach will enable printers to become fully independent operators capable of detecting and correcting errors in real time.

[0059] 1 shows a flow chart of exemplary steps for closed-loop control of a manufacturing process. The method may be performed by an apparatus (or a component thereof) used to perform the manufacturing process. Alternatively, the method may be performed by a remote server remote from the apparatus used to perform the manufacturing process.

[0060] The method begins by receiving at least one image of the manufacturing process at predefined time intervals during the manufacturing process (step S100).

[0061] The step of receiving at least one image at a predefined time interval (S100) may comprise receiving at least one image at a predefined time interval between 1 second and 10 seconds. In some cases, the at least one image may be received at a predefined time interval of less than 1 second. For example, an image sensor may be used to capture images at a rate of 30 frames per second (30 fps). It will be appreciated that these are exemplary, non-limiting predefined time intervals and that any suitable time interval may be used.

[0062] It may be useful to receive at least one image at predefined time intervals throughout the manufacturing process such that the overall duration of the manufacturing process is monitored and controlled. However, often times, when an error (i.e., deviation of at least one manufacturing parameter) occurs at the beginning of the manufacturing process, it may have a negative impact on the rest of the process if the error is not corrected or is not correctable. Therefore, it may be useful to monitor the beginning of the manufacturing process (i.e., an initial part thereof) since if an error occurs at this stage, it may be more efficient in terms of cost, time, energy, and materials to correct the at least one parameter or to stop the process. Thus, receiving at least one image at predefined time intervals may comprise receiving at least one image at predefined time intervals during at least an initial part of the manufacturing process.

[0063] The method comprises processing at least one image in each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process (step S102). Step S102 may comprise processing the at least one image using a classification module of the trained ML model to predict a value of the at least one manufacturing parameter.

[0064] The method comprises determining whether a predicted value of at least one manufacturing parameter is within a predetermined range of values ​​(step S104).

[0065] If, in step S104, it is determined that the predicted value is within the predetermined range of values, the method returns to step S100.

[0066] If in step S104 it is determined that the predicted value is not within a predefined range of values ​​for that parameter, the method comprises generating instructions for corrective action when the predicted value is outside the predefined range of values ​​(step S106).

[0067] The step of generating instructions for corrective action (S106) may comprise generating instructions for adjusting a value of at least one manufacturing parameter. This may be useful when at least one manufacturing parameter deviates outside a predefined range of acceptable values, yet the manufacturing process is not yet adversely affected. For example, when the manufacturing process involves 3D printing an object, if the object is not adversely affected or damaged by the deviation of the at least one manufacturing parameter, it may be useful to correct / adjust the parameter and continue 3D printing the object. In this case, the method may comprise receiving a confirmation that the value of the at least one manufacturing parameter has been adjusted, and processing the at least one image received after the confirmation is received using the trained machine learning ML model.

[0068] Alternatively, the step of generating instructions for corrective action (S106) may comprise generating instructions to stop the current manufacturing process. This may be useful when deviations of at least one manufacturing parameter outside a predefined range of acceptable values ​​cause the manufacturing process to be adversely affected. For example, when the manufacturing process involves 3D printing an object, it may not be useful to continue 3D printing the object when the object is adversely affected or damaged by deviations of at least one manufacturing parameter. Instead, it may be efficient in terms of cost, time, energy and materials to stop 3D printing the object. The manufacturing process may be restarted from the beginning.

[0069] In some cases, the method shown in FIG. 1 may be performed in real time to enable real-time control of the manufacturing process. In these cases, corrective actions are performed in real time or near real time with respect to the current iteration of the manufacturing process. This may be possible if the trained machine learning model is part of or near the equipment used to perform the manufacturing process, for example.

[0070] Alternatively, the method of Fig. 1 may be performed after a current iteration of the manufacturing process has finished, to allow control of subsequent iterations of the manufacturing process. Performing the method after the current iteration of the manufacturing process has finished means that the entire manufacturing process can be analyzed and instructions for corrective actions can be issued to subsequent iterations of the manufacturing process. The error detection process may also be applied as a measure for quality control measures after the manufacturing process has finished. This may be particularly useful, for example, in the production of medical devices.

[0071] If the method is not performed in real time, the corrective action is performed on a subsequent iteration of the manufacturing process. This may be useful when the trained machine learning model is, for example, not part of or near the equipment used to perform the manufacturing process. For example, if the trained ML model is remote from the equipment (e.g., located on a remote server or cloud server), the time to send images to the remote server and send instructions back to the equipment for corrective action may be too long for the manufacturing process to be effectively controlled in real time or near real time. Therefore, it may be more useful to use information received for one iteration of the manufacturing process to control another subsequent iteration. This may also be useful when the corrective action is to stop the current iteration of the manufacturing process. In another example, an error in at least one manufacturing parameter may intensify over time (e.g., during an iteration of the manufacturing process) and / or the parameter may not be correctable in real time. For example, errors such as cracking and warping deformations, where stresses in the object being manufactured build up over time, cannot be corrected in real time. In this case, corrective action can only be taken with respect to subsequent iterations of the manufacturing process.

[0072] FIG. 2 is a block diagram of a system 200 and an apparatus 100 for closed-loop control of a manufacturing process.

[0073] The apparatus 100 is for performing a manufacturing process using closed-loop control. The apparatus comprises at least one processor 102 coupled to a memory 104. The at least one processor 102 may comprise one or more of a microprocessor, a microcontroller, and an integrated circuit. The memory 104 may comprise volatile memory, such as a random access memory (RAM) for use as temporary memory, and / or non-volatile memory, such as flash, read-only memory (ROM), or electrically erasable programmable ROM (EEPROM), for example, to store data, programs, or instructions.

[0074] The apparatus 100 may include a trained machine learning ML model 106.

[0075] The processor 102 is configured to receive at least one image of the manufacturing process at predefined time intervals during the manufacturing process, process the at least one image at each time interval using the trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process, determine whether the predicted value of the at least one manufacturing parameter is within a predefined range of values, and generate instructions for corrective action when the predicted value is outside the predefined range of values.

[0076] The apparatus 100 may further comprise at least one image capture device 108 for capturing at least one image of the manufacturing process at a predefined time interval. The image capture device 108 may be any one of a camera, a light sensor, and an infrared sensor or camera.

[0077] The apparatus 100 may be any one of an extrusion-based 3D printer, an additive manufacturing apparatus, a material extrusion apparatus, a stereolithography apparatus, a laser powder bed fusion apparatus, a milling apparatus, a turning apparatus, a lathe, a laser cutter, and a plasma cutter, it being understood that this is a non-exhaustive list of possible apparatus.

[0078] As mentioned above, the techniques may be performed in real time to enable real-time control of the manufacturing process. In these cases, corrective actions are performed in real time or near real time with respect to the current iteration of the manufacturing process. This may be possible if the trained machine learning model 106 is part of or near the equipment 100 used to perform the manufacturing process, for example.

[0079] Alternatively, as described above, the method may be performed after a current iteration of the manufacturing process is completed to allow control of subsequent iterations of the manufacturing process. In these cases, corrective measures are performed with respect to subsequent iterations of the manufacturing process. This may be useful when the trained machine learning model is, for example, not part of or near the apparatus 100 used to perform the manufacturing process. In this case, the trained machine learning model may not be part of or may not be stored on the apparatus 100. Instead, the apparatus 100 may transmit data to a trained machine learning model located remotely from the apparatus 100.

[0080] Thus, the device 100 may comprise a communication module 110 for transmitting at least one image for processing.

[0081] The system 200 may include a remote server 112. The remote server 112 may include at least one processor 114 coupled to a memory 116. The remote server 112 may include a trained ML model 118. The at least one processor 114 may be configured to receive at least one image of the manufacturing process from the apparatus 100, process the at least one image at each time interval using the trained machine learning ML model 118 to predict a value of at least one manufacturing parameter associated with the manufacturing process, determine whether the predicted value of the at least one manufacturing parameter is within a predetermined range of values, and generate instructions for corrective action when the predicted value is outside the predetermined range of values.

[0082] The at least one processor 114 may be further configured to transmit the generated instructions to the device 100 .

[0083] In some cases, the steps performed by the at least one processor 114 may be performed in real time and the generated instructions may be transmitted while the manufacturing process is ongoing.

[0084] The step of generating instructions for corrective action may comprise generating instructions for adjusting a value of the at least one manufacturing parameter, in which case the at least one processor 114 may be further configured to receive a confirmation from the apparatus 100 that the value of the at least one manufacturing parameter has been adjusted, and to process the at least one image received after the confirmation is received using the trained machine learning ML model 118.

[0085] Alternatively, the step of generating instructions for corrective action may comprise generating instructions to stop the current manufacturing process.

[0086] The steps performed by the at least one processor 114 may be performed after a current iteration of the manufacturing process has finished and the generated instructions may be transmitted before a subsequent iteration of the manufacturing process begins, in which case the step of generating instructions for corrective action may comprise generating instructions for adjusting a value of at least one manufacturing parameter of the subsequent iteration of the manufacturing process.

[0087] The technique will now be described with respect to a particular type of manufacturing process, namely extrusion-based 3D printing. It will be understood that the following illustrates, but is a non-limiting example of, how the technique may be used. Furthermore, while four manufacturing parameters are taken into account in this example, it will be understood that these are non-exhaustive and non-limiting parameters.

[0088] Dataset creation, filtering, and augmentation 3A-3F show an overview of the CAXTON system used for automated data collection.

[0089] A network of eight FFF 3D printers was used for data collection. Creality CR-20 Pro printers were chosen due to their low cost, pre-installed bootloader, and included Z-probe. Firmware for each printer was flashed to Marlin 1.1.9 to ensure that thermal runaway protection was enabled, which is essential for the printers to be left unattended. The standard firmware configuration process enabled EEPROM chit chat, as well as a new axis range for the bed remover. Each printer was then outfitted with a Raspberry Pi 4 Model B, which acts as a networking gateway to send and receive data to and from the printer via serial. The Pi runs a Raspbian-based distribution of Linux, and the OctoPrint server with custom developed plugins. A low-cost consumer USB webcam (Logitech C270) was connected to the Pi to take snapshots. The camera was mounted facing the nozzle tip using a single 3D printed part. These components can be easily adapted to new and existing printers at minimal cost, and thus are believed by applicants to be the first truly scalable and deployable system of its kind.

[0090] The printer used for direct inking was a modified Creality Ender 3 Pro. The extruder setup was designed and built in-house and utilized a stepper motor driven syringe with a Luer lock nozzle. The printer was equipped with a Pi, a Z-probe, and a Raspberry Pi Camera v1 with a zoom lens. The firmware was a modified version of Marlin 2.0.

[0091] FIG. 3A shows a workflow for collecting various data along with automatic labeling of images with 3D printing parameters. A new 3D printing dataset was generated, including parts printed using polylactic acid (PLA), and labeled with their associated printing parameters for a wide variety of shapes and colors in a fleet of extrusion-based 3D printers. The data generation pipeline disclosed in this application automates the entire process from STL file selection to tool path planning, data collection, and data storage (see FIG. 3A). Model geometries were automatically downloaded randomly from the online repository Thingiverse.

[0092] FIG. 3B shows how images are captured using a fleet of eight FFF 3D printers equipped with image capture devices (e.g., cameras) focused on the nozzle tips to monitor extrusion. During printing, images are captured every 0.4 seconds. Each captured image is time-stamped and labeled with the current printing parameters, i.e., actual and target temperatures for the hot end and bed, flow rate, lateral speed, and Z offset. Additionally, the nozzle tip coordinates on each printer are stored for each image to allow for easy cropping around the area of ​​interest during training.

[0093] FIG. 3C shows a rendering of the generated toolpath for a single input shape with randomly selected slicing parameters.

[0094] Figure 3D shows snapshots of data collected during a print showing images with various parameter combinations. After 150 images are collected, a new combination of printing parameters is generated for every printer by sampling a uniform distribution of each parameter. The new parameter combination is sent as a gcode command over the network to each printer, which is then executed. At runtime, another 150 labeled images are collected before the parameter update process occurs again. This continues until the end of the print, resulting in a set of images each with very different printing parameters (see Figure 3D).

[0095] A new and simple method for removing completed prints is developed so that the printer can continue to operate without human intervention. Numerous methods have been implemented previously to automatically remove parts upon completion, but previous implementations either require extensive hardware modifications, which are costly, or only allow for a relatively limited range of parts to be removed. FIG. 3E shows a bed remover and dock design that utilizes existing motion systems with photos taken during operation. Here, a new simple and effective bed removal system is proposed without the need for additional electronics, motors, or complex mechanical parts. The proposed solution can be retrofitted to any extrusion printer and consists mainly of the printed parts that can be produced by the printer in question. The already moving print head moves with a scraper located behind the build platform and enters the dock. The printer's built-in motors are then used to move the print head and scraper across the build surface to remove the printed object. After removal, the print head returns the scraper to its home location and exits the dock. To ensure the scraper always stays in the same position, a scraper and dock with a magnet is attached to the print bed, holding the scraper in place until the next object requires removal.

[0096] Figure 3F shows the normalized parameter distribution in the entire dataset collected by CAXTON, which contains over 1.2 million samples. Due to sampling suboptimal parameter combinations, some prints are complete failures and provide little information about the relevant parameters after a certain point. Such images are manually removed, leaving 1,166,552 labeled images (91.7% of the original 1,272,273). The remaining dataset contains somewhat noisier labels due to the longer response time in updating printing parameters such as flow rate before there is a noticeable change in the image. The response time consists of command execution delay and mechanical delay. The former delay is mostly handled by only capturing the image after the acknowledgement response of the parameter update command is received from the printer. Worst case experiments were performed to determine the response time for changing each parameter from minimum to maximum in the dataset against mechanical delay. It was found that the change was mostly visible within 6 seconds of the update being applied, and therefore 15 images are removed after the parameter update. This leaves 1,072,500 samples where the system has reached its desired state. Unrealistic parameter outliers, caused by the printer not executing gcode commands correctly or glitches in sensors such as thermistors, are filtered out, leaving 991,103 samples. Finally, very dark images with average pixel values ​​across less than 10 RGB channels are removed. This results in a clean dataset consisting of 946,283 labeled images (74.4% of the original images). The now continuous parameter values ​​are then binned into three categories for each parameter: low, good, and high. The upper and lower limits for these bins are selected empirically. This creates 81 possible different class combinations (three categories for four parameters) for the network to predict.

[0097] In deep learning, large datasets are preferred to avoid overfitting and to achieve a high level of generalizability, therefore data augmentation is used to further increase the size and quality of the filtered dataset, which improves the performance of the trained model. The locality and type of deposited material in the captured images vary significantly depending on the geometry of the printed model. Additionally, color, reflectance, and shading were all found to vary with camera position, material selection, and printer mechanics. Thus, to increase the size of the dataset, each image was subjected to a wide variety of data augmentation techniques to simulate a wider variety of shapes, camera locations, and materials. First, the full-sized image captured by the camera is randomly rotated by up to 10 degrees in either direction. Then, a slight perspective transformation is applied with a probability of 0.1. In the next step, the image is cropped to a square region of 320x320 pixels focused on the nozzle tip using the coordinates stored during data collection. The rotation and perspective transformation are applied before the crop to practically eliminate the need for padding within the cropped region. A random square portion of the 320x320 image with an area between 0.9 and 1.0 is then cropped and resized to 224x224 pixels, i.e., the input size for the deep neural network. A horizontal flip may then be applied to the image with a probability of 0.5, followed by a ±10% jitter to the image's brightness, contrast, hue, and saturation. Finally, the channels in the transformed image are normalized using the pixel mean and standard deviation of each channel for all images in the filtered dataset.

[0098] Model architecture, training, and performance FIG. 4A shows an exemplary architecture of a neural network used to perform the closed-loop control of the present technique. In this case, the architecture comprises an attention module and a residual block with a separate fully connected output branch for each parameter. The attention module consists of a trunk branch including the residual block, and a mask branch performing downsampling and upsampling. It will be understood that other suitable architectures may be used to perform the closed-loop control described herein. For example, the neural network may comprise a convolutional layer (and may be based on, for example, ResNet, EfficientNet, RegNet, or ConvNext) and / or a transformer (for example, ViT or BiT). More generally, as described above, the neural network may comprise any or all of a convolutional layer, a skip connection, an attention layer and an attention mask, and a densely connected layer.

[0099] Accurate prediction of the current printing parameters in the extrusion process from an input image is achieved using a residual attention network in Figure 4A with a single backbone and four-headed output branches, one per parameter. The use of attention reduces the number of network parameters required to achieve the same performance on a standard image classification dataset while making the network more robust against noisy labels. Additionally, the attention map in the network allows a level of transparency to detect errors as well as help explain predictions. The shared backbone allows feature extraction to be shared for each parameter, thus reducing inference time compared to having separate networks. Additionally, it allows the network to model interactions between different parameters. Each branch has three output neurons to classify the parameter as low, good, or high. With this structure, the network simultaneously predicts the flow rate, lateral velocity, Z offset, and hot-end temperature states from a single RGB input image. This can be exceptionally challenging for a skilled human operator, although the final trained one classifies the state of all these parameters in our various test sets, achieving a high classification accuracy of 84.3% (averaged over the four parameters). This is particularly difficult when many of the parameters are interdependent, and having a higher Z offset, where the nozzle is farther from the bed, can easily be mistaken for having a low flow rate and under extrusion. Thus, accuracy is not a perfect metric for determining the effectiveness of a network, since in real-world deployments, several different combinations of measures may lead to good extrusion. For each parameter, the following classification accuracy was obtained for the test set: flow rate 87.1%, transverse speed 86.4%, Z offset 85.5%, and hot end temperature 78.3%.

[0100] The network is mainly composed of three attention modules and six residual blocks, and is based on the Attention-56 network. The attention module is composed of two branches, namely the mask branch and the trunk branch. The trunk branch performs feature processing of the conventional network and is constructed from the residual block. The mask branch undertakes downsampling followed by upsampling to learn an attention mask used to weight the output features of the module. This mask can be used not only during the forward pass for inference, but also as a mask in the backward pass during backpropagation. This was one of the reasons for choosing this network architecture, since the mask branch can make the network more robust to noisy labels that the dataset contains due to parameter changes and fine discrepancies during printing. After these blocks, the network is flattened into a fully connected layer that links each of the separate four branches. The branches need to be separate outputs of the network, since each prediction requires that each branch has a single prediction.

[0101] FIG. 4B shows the confusion matrix of the final network for each parameter in the test dataset, namely, flow rate, lateral velocity, Z offset, and hot end temperature.

[0102] The ML model may be trained using a single stage or N stages. For the particular architecture described above, it was found that splitting the training process into three separate stages was most robust. This exemplary training method is described below, but it will be understood that this is merely exemplary and that other suitable training methods may be used with this or a different architecture.

[0103] 5A-5D show three stages of training a residual attention CNN with the CAXTON 3D printing parameter dataset. FIG. 5A shows training and validation accuracy plots for training the network across three seeds smoothed with an exponential moving average for three datasets: the single-layer dataset, the entire dataset, and the balanced dataset. FIG. 5B shows the parameter-wise validation accuracy plots and their joint average for the best of the three seeds smoothed with an exponential moving average. FIG. 5C shows the learning rate decay for each training across the three seeds using a drop in plateau learning rate scheduler. The learning rate drop results in a noticeable improvement in accuracy at several points in FIG. 5A and FIG. 5B. FIG. 5D shows the initial learning rate for each training stage, chosen by sweeping a wide learning rate range and selecting a value with a steep drop in loss.

[0104] For each stage, a differently seeded network was trained. In the first stage, the network is trained on a sub-dataset that contains only images from the first layer with 100% infill. Features are more visible for each parameter in these prints, and by first training with this subset, the network can learn to detect important features more quickly. It was found that this separation speeded up the learning process, since features were more learnable for a single layer, and could then be trained on the entire dataset, making the network generalizable to complex 3D shapes. The best seed achieved a training accuracy of 98.1% and a validation accuracy of 96.6%. A transfer learning technique was then used to retrain the best seed's model on the entire dataset, which contained images for all 3D shapes. This was done three times, with the best seed achieving a training accuracy of 91.1% and a validation accuracy of 85.4%, respectively. Networks can learn inherent biases in the data given to them, so due to imbalance in the overall dataset (e.g., Z offset can have a much larger value that is too high than too low because the nozzle will hit the print bed), transfer learning was used again. However, this time only the final fully connected layer for each of the four branches was trained on a balanced sub-dataset containing an equal number of samples for each of the 81 possible combinations (four parameters, each of which can be low, good, or high). The remainder of the network weights were frozen. This achieved a training accuracy of 89.2% and a validation accuracy of 90.2%.

[0105] The final trained network was tested against a test set consisting of random samples from the entire shape dataset, achieving an accuracy of 84.3%.

[0106] To train the network, the cross-entropy loss at each of the branches was determined, and these losses were then summed together before backpropagation. This results in a "shared" region of the updating network to accommodate, for each branch, connections to the branch being updated with only its own loss. The initial learning rates were selected in each of the three training stages by sweeping a large range of values ​​and choosing a learning rate with a large drop in loss. The learning rates for each of the stages can be seen in FIG. 5D. The selection of the appropriate learning rate was critical; a learning rate that was too high led to poor attention maps, while a learning rate that was too low resulted in the network taking too long to train or getting stuck in an early local minimum. The AdamW optimizer was used during training with a drop in the plateau learning rate scheduler to reduce the learning rate by a factor of 10 when the loss did not improve by more than 1% after three consecutive epochs. A plot of the learning rate during training can be seen in FIG. 5C. Train, validation, and test splits of 0.7, 0.2, and 0.1, respectively, were used with a batch size of 32. Three stages of training were trained for 50, 65, and 10 epochs, respectively. Each stage was trained three times with three different seeds. During transfer learning, the best seed from the previous stage was chosen as the basis for continuing training from there.

[0107] To visualize which features the network focuses on at each stage, images of the attention map after each module were created as seen in Figure 5B. Here, the same attention mask from each module is applied to each of the three input images with non-target areas darkened (Note: these masks are illustrative examples, as each module contains many different attention maps). The network focuses explicitly on the printed area in the example mask output for attention module 1, and then only on the most recent extrusion for module 2. Module 3 applies the opposite, focusing on everything except the nozzle tip before. To provide a visual illustration, Gradient-weighted Class Activation Mapping (Grad-CAM) was also used, which can be thought of as a posteriori attention. It was found that for all parameter combinations, the network focuses primarily on the most recent extrusion from the nozzle. This is good for fast response times and corrections, and it is desirable for the network to use the most recently deposited material for its predictions.

[0108] Online Calibration and Parameter Discovery Pipeline 6A-6C show a machine vision control system pipeline and feedback parameters. It will be understood that these figures show an example pipeline and example feedback parameters. The feedback parameters and their values ​​may vary and may be adjusted depending on the manufacturing process being controlled.

[0109] Figure 6A shows six key steps in the feedback pipeline that allows online parameter updates from images of the extrusion process. Figure 6B shows a table including θmod (mode threshold), L (series length), Imin (interpolation minimum), A+ (maximum increase), A- (maximum decrease) for each printing parameter along with possible level updates. Figure 6C shows a simple example single layer shape showing tool path division into smaller equal segments. To allow for rapid corrections, a length of 0.5 mm is used in the feedback process, which dramatically increases the gcode file size.

[0110] Once downloaded, each 3D model was sliced ​​using different settings for scale, rotation, infill density, number of perimeters, and number of solid layers by randomly sampling from a uniform distribution with an infill pattern chosen from a given list of shared patterns. The generated set of toolpaths is then transformed to have a maximum movement of 2.5 mm using a custom script to allow faster response time for parameter changes during printing. During the printing process, images of the nozzle tip and material deposition are taken at 2.5 Hz and sent to a local server for inference. Each received image is cropped to a 320x320 pixel region focused on the nozzle tip. The user needs to specify the pixel coordinates of the nozzle once mounting the camera in the setup. Additionally, the user may wish to vary the size of the cropped region depending on the camera position, focal length, and size of the printer nozzle. Choosing a suitable region around the nozzle affects the performance of the network, and the best balance between accuracy and response time is found when approximately five extrusion widths are visible on either side of the nozzle tip.

[0111] The cropped image is then resized to 224x224 pixels and normalized across RGB channels. Given this image as input, the classification network then produces a prediction for each parameter. These predicted parameters are stored in separate lists of different set lengths L for each parameter, and the length of these lists was the key variable to adjust to balance response time with accuracy (see Figure 6B). The lengths of these lists were determined by running experiments for each parameter separately in the same printing conditions. Larger list lengths result in more accurate predictions but slower response times, and vice versa for smaller list lengths.

[0112] When the list is complete, a mode threshold θmode is used to determine the resulting prediction. This threshold is another variable that is tuned for each parameter by performing experiments for each parameter. If no mode is found, no updates are made and the printing parameters are treated as being "okay". If a mode is found, the size of the mode (a ratio of the list length) is used to scale the response.

[0113] The modal ratio was used to scale the amount by which a given parameter should be updated. However, when the modal ratio is larger than the threshold, there is little room for feedback adjustment. Therefore, a one-dimensional linear interpolation is applied to scale the modal ratio to a larger range. This interpolation maps the range between the parameter threshold and 1 to a new minimum Imin and 1 adjusted for each parameter. This interpolated minimum for each parameter is another variable that is adjusted using individual experiments. The interpolated ratio is then used to scale the update amount for each parameter, i.e., the increment A. + and the reduction amount A -are used as scale factors to adjust both the maximum positive and negative update amounts. The maximum positive and negative update amounts are more variables that are adjusted for each printing parameter. The final values ​​for all these variables can be seen in Figure 6B, and these values ​​were obtained iteratively via experimentation for each parameter individually.

[0114] Once the final updates have been calculated for the printing parameters, they are sent to the Raspberry Pi attached to each printer. The Pi retrieves the current value for each parameter and then uses the received updates to create the desired gcode command to update the parameters to their new values. The Pi then tries to find an acknowledgement of the execution of the commands by the firmware over serial. Once all commands have been executed by the firmware, the Pi sends an acknowledgement to the server. Once the server receives an acknowledgement that all updates have been executed, it starts making predictions again. Waiting for this acknowledgement of all parameter updates is essential to stop oscillations caused by over- or under-shooting targets, and making predictions is only desirable after updates have been applied.

[0115] To prove the efficacy of the system in correcting the printing parameters, each was tested individually. The same model printer as in training was used, but with an altered camera position, a different 0.4 mm nozzle, and a first-look single-layer printing sample. To compare responses between parameters, the same printer and conditions were used for each experiment, and each was printed using the same spool of PLA filament (PLA was the material used for all training data).

[0116] Figures 7A-7F show printer and feed material agnostic online parameter correction and discovery. Figure 7A shows the in-situ rapid correction of a manually induced erroneous single parameter using a trained single residual attention CNN model printed with PLA feed material on a known printer with a virgin nozzle that was not used in the training data. Figure 7B shows the online simultaneous optimization of multiple improper parameters in a virgin thermoplastic, demonstrating that the control pipeline is robust to a wide variety of feed materials with different material properties, colors, and initial conditions. Figure 7C shows that, mimicking a human operator, the system uses self-learned parameter relationships for correction prediction. A high Z-offset can be fixed by both decreasing the Z-offset and / or increasing the material flow rate. Figure 7D shows the setup transfer to other extrusion processes, such as direct inking of PDMS on completely virgin hardware (camera, printer, nozzle, extrusion process, material, etc.). Figure 7E shows that the correction of multiple improper printing parameters introduced mid-prints. Both rooks were printed in the same conditions with the only difference being the compensation. Figure 7F shows the compensation of a print that started with an incorrect parameter combination. All six spanners were printed in the same conditions.

[0117] Manually induced errors for each parameter were added to the file and the control loop response was recorded. An experimental pipeline was built to take the input STL file, slice it with sensible print settings, insert gcode commands to change parameters to low values, then parse the generated gcode and split the model into 1.0mm sections. Splitting the gcode reduces firmware response times that can be very long. It was found that 1.0mm allowed for quick corrections while keeping the gcode file size manageable. Attempting corrections without toolpath splitting was incredibly slow for some shapes, especially those with long moves. Splitting the gcode file into chunks smaller than 1.0mm limited print speed and introduced jitter. This was caused by the printer being unable to read and process the gcode lines fast enough. Figure 7A demonstrates the corrections for each of the parameters along with the parameters over the illustrated time along with the printed parts and predictions. The impact of poor manually induced printing parameters can be easily seen for the flow rate, Z offset, and hot end. For the lateral speed, upon closer inspection, darker lines can be seen that are located only around the slower print speeds. Note the small delay from when the command is sent to when it prints, indicated by the black arrow, and when the parameters update their values. This shows the importance of waiting for the recognition response along with the benefit of toolpath segmentation. The prediction plot demonstrates how effective the network after mode thresholding is in predicting the proper printing state. Due to the time it takes to cool and heat up, along with the longer list of predictions and the need for larger mode thresholds for safety reasons, the hot end response time is significantly longer than the other printing parameters, and a hot end temperature increase should therefore only be implemented when it is reasonably certain that this is required.

[0118] The control pipeline generalizes to virgin thermoplastic feed materials in a variety of colors with a wide variety of different material properties. Figure 7B shows the online correction of multiple parameters for four different thermoplastics. Each of these samples was started with a different combination of multiple inappropriate printing parameters. The TPU and carbon fiber filled sample does not have a printed perimeter due to poor initial conditions. However, multiple parameters are successfully updated per network resulting in good extrusion. This is not only useful for automated parameter discovery to help users in tuning their printer for new materials by quickly obtaining the best parameter combination, but also shows that the control system can improve productivity by eliminating failing prints where the initial toolpath does not adhere to the bed. Thanks to having all the parameter predictions in one network structure, the trained model can learn the interactions between multiple parameters and give creative solutions to inappropriate parameters in the same way that a human operator would. The sample was printed using the control loop setup but without adding online corrections. This sample contained an area with a high Z offset. A high Z offset results in separate paths of extruded material, and the same result can occur during extrusion. Figure 7C shows the network determining that increasing the flow rate along with decreasing Z will result in good extrusion. When a trained model can find multiple ways to solve the same problem, it can be more robust to improper predictions for a single parameter, and allow for faster feedback by combining updates across multiple parameters. The prediction plot also shows how quickly the network realizes that the parameters are now good, which is essential to ensure that the control system does not overshoot when applying online corrections.

[0119] To further test the developed control pipeline to its limits, it was tested against a different printer modified for direct inking of PDMS. The direct inking system uses a stepper motor with a threaded rod to move a plunger inside the syringe. Various models of cameras were also used, mounted at different positions with a transparent and reflective print bed made of glass. The nozzle size of the direct inking system was also different at 0.24 mm. For this test with PDMS printed at room temperature, only the flow rate was adjusted. Figure 7D shows that the network learns to increase the flow rate and pressure to print the material. It was found that the flow rate correction in one direction would stop once the set pressure was reached. It was found that sometimes during long prints, the control loop would overshoot the flow rate if the flow rate correction was not stopped fast enough due to a large increase in pressure inside the syringe. However, this problem is specific to the combination of syringe, small diameter nozzle, and material selection. When printing other materials, this overshoot and pressure delay became less of an issue. All previous parameter corrections in the single-layer shape have been done on the final network trained on the 3D shape. Therefore, the network also works on the entire 3D model, and the single-layer shape was chosen to easily and visually show the corrections of each parameter. In Figure 7E, the control pipeline was used on a wide range of entire 3D shapes to demonstrate that the method can be used in a production setting. Each of these prints was started with an inappropriate parameter combination.

[0120] Various network structures can be applied to the collected data sets to improve control loop performance, provide quality control metrics, perform fast error detection, and even predict mechanical performance.

[0121] While the above describes what is considered to be the best mode, and where appropriate other modes of carrying out the present technique, those skilled in the art will appreciate that the present technique should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognize that the present technique has a wide range of applications, and that numerous and varied modifications may be made to the present embodiment without departing from any inventive concept as defined in the appended claims. [Explanation of symbols]

[0122] 100 devices 102 processors 104 Memory 106 trained ML models 108 Image Capture Device 110 Communication Module 112 Remote Server 114 processors 116 Memory 118 trained ML models 200 Systems

Claims

1. 1. A computer-implemented method for closed-loop control of a manufacturing process, comprising: receiving at least one image of the manufacturing process at predefined time intervals during the manufacturing process; processing the at least one image at each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter related to the manufacturing process; determining whether the predicted value of the at least one manufacturing parameter is within a predetermined range of values; generating instructions for corrective action when the predicted value is outside of the predetermined range of values; A method comprising:

2. The method of claim 1 , wherein generating instructions for corrective action comprises generating instructions for adjusting a value of at least one manufacturing parameter.

3. receiving a confirmation that the value of the at least one manufacturing parameter has been adjusted; processing at least one image received after the confirmation is received using the trained machine learning ML model; The method of claim 2 , further comprising:

4. The method of claim 1 , wherein generating instructions for corrective action comprises generating instructions to stop a current manufacturing process.

5. 5. The method of claim 1, 2, 3, or 4, wherein receiving at least one image at a predefined time interval comprises receiving at least one image at a predefined time interval between 0 and 10 seconds.

6. 6. The method of claim 1, wherein receiving at least one image at a predefined time interval comprises receiving at least one image at a predefined time interval during at least an initial portion of the manufacturing process.

7. sending a command to pause the manufacturing process; performing said processing, determining, and generating steps while said manufacturing process is paused; 7. The method of claim 1, further comprising:

8. 8. The method of claim 1, wherein processing the at least one image using a trained machine learning ML model comprises processing the at least one image using a classification or regression module of the trained ML model to predict a value of the at least one manufacturing parameter.

9. 9. The method according to claim 1, wherein the method is performed in real time to allow real time control of the manufacturing process.

10. 9. The method of claim 1, wherein the method is performed after the manufacturing process has finished to allow control of subsequent iterations of the manufacturing process.

11. 11. The method of any one of claims 1 to 10, wherein the manufacturing process is an extrusion-based 3D printing process.

12. 12. The method of claim 11, wherein the at least one production parameter is any of a flow rate, a transverse speed or feed rate, a Z-axis offset, a hot end temperature, a bed temperature, a layer height, a line width, an infill density, a wall thickness, and a retract setting.

13. obtaining a training dataset comprising a plurality of images of the manufacturing process, each image being labeled with a plurality of manufacturing parameters associated with the manufacturing process and a timestamp; inputting images from the training dataset into the ML model; processing an input image with the ML model to identify one of the manufacturing parameters; predicting a value for each manufacturing parameter for the input image using the ML model; comparing the predicted value with the label of the image; and updating the ML model to reduce the difference between the predicted value and the label of the image. The method of claim 1 , wherein the trained ML model is trained.

14. A non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to perform the method of any one of claims 1 to 13.

15. 1. An apparatus for conducting a manufacturing process using closed loop control, comprising: The apparatus comprises at least one processor coupled to a memory; The at least one processor: receiving at least one image of the manufacturing process at predetermined time intervals during the manufacturing process; processing the at least one image at each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process; determining whether the predicted value of the at least one production parameter is within a predetermined range of values; generating instructions for corrective action when the predicted value is outside the predetermined range of values; 4. An apparatus configured to:

16. 16. The apparatus of claim 15, further comprising at least one image capture device for capturing the at least one image of the manufacturing process at predefined time intervals.

17. 17. The apparatus of claim 15 or 16, wherein the apparatus is any one of an extrusion-based 3D printer, an additive manufacturing apparatus, a material extrusion apparatus, a stereolithography apparatus, a laser powder bed fusion apparatus, a milling apparatus, a turning apparatus, a lathe, a laser cutter, and a plasma cutter.

18. 1. A system for closed loop control of a manufacturing process, comprising: The system comprises: An apparatus for carrying out the manufacturing process, comprising: at least one image capture device for capturing at least one image of said manufacturing process at predefined time intervals; and a communication module for transmitting the at least one image for processing. An apparatus comprising: a remote server having at least one processor coupled to a memory; Equipped with The at least one processor: receiving the at least one image of the manufacturing process from the device; processing the at least one image at each time interval using a trained machine learning ML model to predict a value of at least one manufacturing parameter associated with the manufacturing process; determining whether the predicted value of the at least one production parameter is within a predetermined range of values; generating instructions for corrective action when the predicted value is outside of the predetermined range of values; configured to system.

19. the at least one processor: Transmitting the generated instructions to the device.

20. The system of claim 18, further configured to:

20. 20. The system of claim 19, wherein the steps performed by the at least one processor are performed in real time and the generated instructions are transmitted while the manufacturing process is ongoing.

21. 21. The system of claim 20, wherein generating instructions for corrective action comprises generating instructions for adjusting a value of at least one manufacturing parameter.

22. the at least one processor: receiving a confirmation from the device that the value of the at least one manufacturing parameter has been adjusted; processing at least one image received after the confirmation is received using the trained machine learning ML model; 22. The system of claim 21, further configured to:

23. 21. The system of claim 20, wherein generating instructions for corrective action comprises generating instructions to stop a current manufacturing process.

24. 20. The system of claim 18, wherein the steps performed by the at least one processor are performed after the manufacturing process has ended and the generated instructions are transmitted before a subsequent iteration of the manufacturing process begins.

25. 25. The system of claim 24, wherein generating instructions for corrective action comprises generating instructions for adjusting a value of at least one manufacturing parameter of the subsequent iteration of the manufacturing process.

26. 1. A computer-implemented method for training a machine learning ML model to enable closed-loop control of a manufacturing process, comprising: obtaining a training dataset comprising a plurality of images of the manufacturing process, each image being labeled with a plurality of manufacturing parameters associated with the manufacturing process and a timestamp; Machine learning ML models, inputting images from the training dataset into the ML model; processing an input image with the ML model to identify one of the manufacturing parameters; predicting a value for each manufacturing parameter for the input image using the ML model; comparing the predicted value with the label of the image; and updating the ML model to reduce the difference between the predicted value and the label of the image. training by A method comprising:

27. 27. The method of claim 26, wherein the plurality of images in the training dataset comprises any one or more of an individual image, an individual frame from a video, and a video comprising a plurality of frames.

28. 28. The method of claim 26 or 27, wherein comparing the predicted values ​​comprises comparing the predicted values ​​to the labels of the images to generate a loss function, and training the ML model comprises using backpropagation to train the ML model to reduce the loss function.

29. 30. The method of claim 26, 27, or 28, wherein training the ML model further comprises training the ML model to generate corrective actions to correct errors at inference time.

30. Training the ML model comprises: generating the ML model using a pre-trained network, the pre-trained network being trained using a first training dataset representing a wide range of objects; training the ML model using a second training data set representing a single object or a family of objects; 30. The method of any one of claims 26 to 29, comprising:

Images