Camera-based monitoring system for 3D printing

Abstract

A collection of camera-based monitoring subsystems can be installed in a composites-based additive manufacturing (CBAM) system to autonomously detect layer-by-layer integrity and correct any issues in real time. Multiple cameras are placed at different locations throughout the process. The resulting images are analyzed to determine whether the processed pages are within tolerance based on comparison to the model. Cameras can be placed on the print platen, the rear conveyor that carries the powder pages to the stacker, and the stacker to ensure the pages are properly stacked and aligned. The invention provides quality assurance and quality control to verify the build process at the layer level. This reduces post-build work and even reveals internal part data, significantly reducing the amount of destructive testing required.

Description

[Technical field]

[0001] This application is related to and claims priority to U.S. Provisional Patent Application No. 62 / 965,096, which incorporates by reference the entire disclosures contained therein and the following U.S. patents: 9,393,770; 9,776,376; 9,827,754; 9,833,949; 10,046,552; 10,252,487; 10,377,080; 10,377,106; 10,384,437; and 10,597,249. [Background technology]

[0002] The present invention relates to three-dimensional (3D) printing, and in particular to monitoring errors during the 3D process with a camera. DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0003] Composite-Based Additive Manufacturing (CBAM) is a process in which multiple cross sections of a three-dimensional object are printed cross-sectionally onto a substrate sheet (such as carbon fiber) using an inkjet printer or lithography techniques. Printing typically uses a water-based ink solution, although in some embodiments other solutions or inks are used. The substrate is then laid down with a powder such as a thermoplastic, thermoset, metal, or other powder. The powder only adheres to the wetted (printed) parts of the substrate. Excess powder is removed from the sheets, and the sheets are laminated together. The laminate is typically compressed and heated, fusing the powder layers to form the 3D object. Excess solid material can then be removed by polishing, sandblasting, chemical means, or other removal techniques.

[0004] In the CBAM process, a defect in one layer can ruin the entire build, and this process is unique in that it has the ability to correct defective layers during the build, for example by rejecting and reprinting the bad page, and then inserting the corrected page into the object.

[0005] However, the conventional methods cannot detect the defective layer in real time.

[0006] Conventional additive manufacturing systems require manual monitoring to observe the object being built. Conventional systems are equipped with multiple cameras that deliver video of the build process and are monitored manually from a remote location. In some direct metal printing systems, cameras are installed to inspect the melt pool and, if necessary, change process parameters that affect the object being built. Currently, there is no technology that can stop the build of an object that is out of tolerance or repair the object being built by modifying the defective layers.

[0007] It would be highly advantageous to have a system and method that could detect defective pages in real time and initiate a repair process without interrupting the build. Summary of the Invention

[0008] The present invention relates to a camera-based monitoring subsystem that can be installed in a CBAM system to autonomously inspect the integrity of each layer and repair problems in real time. Multiple cameras are placed at different locations throughout the process. The resulting images are analyzed to verify whether the processed pages are within tolerance based on comparison to the model. Cameras are placed on the print platen, the rear conveyor that carries the powder pages to the stacker, and on the stacker to verify that the pages are properly stacked and aligned. The present invention provides quality assurance and quality control (QA / QC) to verify the build process at the layer level. This reduces post-build work to perform QA / QC and shows data down to the internal parts, significantly reducing the destructive testing required. [Brief description of the drawings]

[0009] The following describes the drawings showing the features of the present invention.

[0010] [Figure 1] FIG. 1 illustrates a portion of a typical CBAM process including the camera positions of the present invention. [Diagram 2] FIG. 2 is a diagram showing a platen camera and lighting. [Diagram 3] FIG. 13 is a diagram showing a rear conveyor camera and lighting. [Figure 4] FIG. 1 shows a stacker camera and lighting. [Diagram 5] 1 is a flow chart of a typical corrective printing process performed using a print platen camera. [Figure 6] 1 is a flow chart of the sorting process after powder deposition. [Figure 7] 1 is a flow chart of a modification process.

[0011] These figures are for understanding the present invention, and the scope of the present invention is not limited to what is shown in the figures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present invention relates to a camera-based monitoring subsystem that can be installed in a CBAM system to autonomously inspect the integrity of each layer and correct any problems in real time. These monitoring systems are used for multiple purposes in the CBAM printing process.

[0013] 1) Inspection: Each monitor in the system inspects the parameters of each stage in the printing pipeline and compares them to the ideal process state. If the difference between the two is within a given tolerance, the layer passes the inspection. If the difference between the two is not within a given tolerance, the layer fails the inspection and the system takes corrective action. The corrective action includes the following steps:

[0014] · Adjusting process parameters: to correct the problem (e.g. adjusting the vacuum level to optimise the level of powder deposition on the page). · Reject and reprint of incorrect pages: Rejected pages can be automatically or manually removed and replacement pages reprinted. -Pause the printing process: If a sudden failure occurs, the print job is paused and the operator is notified of the situation.

[0015] 2) Record: The inspection data is recorded to a file, database, or web (or cloud) service. The recorded data includes the full scanned image, a low-resolution image, ancillary calculated data (e.g. how far a measurement is from the tolerance range), and metadata based on the analysis of the build-up. The logged data is used for the following purposes:

[0016] Monitoring: Log data is used for real-time monitoring of running print jobs. Operators can monitor the system from the display, the printer, or remotely using the data transmitted in real time. Quality Control (QC): Logged data can be used to ensure that print conditions are within specified tolerances. Post-mortem verification: Log data can be used for post-mortem verification of the build process. For example, stacking data can provide useful information for part failure analysis. Maintenance: The recorded data is useful for monitoring the operating status of the system for maintenance purposes.

[0017] Detecting problems during the build can save significant time and money. Not only does it reduce the time to complete a scrapped part, it also limits the amount of wasted material. Also, failures during the build can result in catastrophic part failures such as delamination. Monitoring the build process, on the other hand, reduces the risk of defective parts entering the supply chain.

[0018] Multiple camera-based monitoring systems are mounted within the build chamber of a CBAM printer. Each monitor focuses on a specific area of ​​the printer and runs inspection software to inspect the condition of a specific part of the print pipeline. The multiple monitor processes can be attached directly to the main processor or can run as independent units (separate processes) on the main processor or on a dedicated processor. The monitors can communicate via inter-process communication, files, or a network.

[0019] FIG. 1 is a stylized diagram showing a portion of a typical CBAM process. A substrate sheet is fed to a printer platen 106 which depicts each cross-section of a 3D object and prints the cross-section pattern with fluid. For overall illustration, an ink spitting donor 122 and an ink head wiper blade 112 are shown. The printed sheet is then transported to a powder ring unit 104 which lays down powder. The powder only adheres to the printed areas. A first camera C1 102a monitors the printing process. After the powdered page is cleaned of excess powder, it is transported by a reverse flow gate 124 to a rear conveyor 103, which transports it to a stacking unit 116 to be stacked on top of other sheets. A second camera C2 102b monitors the rear conveyor and a third camera C3 102c monitors the stacker. All three cameras transmit images to a processor 114. A rejected page bin 110 receives rejected pages. Rejected pages are either discarded or manually analyzed.

[0020] Each camera monitoring system is registered with the processor 104. When a page is sent during the printing process, the processor notifies each monitoring process, either directly or via broadcast, that the page is being transferred. The message contains the ideal process information (model) and the acceptable tolerance interval of the layers. The monitor then captures the sensor data (e.g., camera images) of the page as it passes in front of the monitoring eye and uses a classification algorithm to measure whether the captured data meets the predetermined tolerance conditions. The monitoring process then notifies the main processor of the classification results and relevant data based on the analysis. The main processor records the data and performs the necessary corrective actions.

[0021] The classification algorithms used within each monitor vary depending on the analysis required for the print stage being inspected. If the page meets the tolerance criteria, a GOOD_PAGE classification is sent to the main process. If the page does not meet the predefined tolerances, a BAD_PAGE classification is sent. The monitoring process in each case also provides sensor data and related data such as measurements and calculations. Table I shows non-limiting examples of various errors, tolerances and parameters monitored by the system of the present invention.

[0022] [Table I]

[0023] The classifier sends a more detailed classification of the problem, if possible. The main process performs subsequent processing (e.g., corrective processing) based on the classification details. Although not shown in FIG. 1, detailed information may be sent regardless of whether the page is classified as good or not. For example, the rate of powder deposition may be estimated from the captured images. If the deposition rate is low, whether it is within an acceptable range or not, the classifier sends data to the main process to increase the powder deposition rate. This data is packaged with the data provided with the classification.

[0024] Typically, monitor classification models use standard image processing techniques to look for image features indicative of specific printing artifacts. Print data for the particular page print under review may (optionally) be used in this classification as an ideal to which the actual print is compared. For example, a thin horizontal line with no powder may indicate a clogged inkjet nozzle, while an oval-shaped missing area in the print may indicate an air bubble in the printer head. A library of sample data representing both good pages and examples of various abnormal conditions is kept for training various machine models such as clustering, regression, and neural network classification models.

[0025] The system according to one embodiment includes three camera-based monitors, as shown in FIG. 1. These run on separate Raspberry Pi™ computers with camera module shields (i.e., daughter boards). The Raspberry Pi is mounted in a sealed case (to prevent ingress of powders and fibers), equipped with a lens and an adjustment screw for adjusting the focus, and has an Ethernet interface to power LED lighting that is connected to and controlled by the Raspberry Pi. The monitoring software is written in Python language and uses several libraries, such as PIL / Pillow, OpenCV (for image processing), NumPy, SciKit, and SciKit-Learn. In addition to the cameras, line scanners or other imaging devices may be used to acquire high-resolution image data.

[0026] The contrast between powdered and non-powdered areas of the page may be increased by illuminating the image with light that emits in the non-visible light spectrum (e.g., UV light) and / or modifying the ink or powder to fluoresce under certain lighting conditions, which is particularly useful when imaging low contrast combinations such as nylon (PA12) polymer powder on glass cloth.

[0027] The three surveillance systems shown in Figure 1 are as follows:

[0028] 1) Print Platen Monitor Cl 102a The monitor monitors the placement of the page on the print platen and can determine whether there is a single page on the platen, whether the page position and angle are within tolerances, whether the registration holes are the correct shape and location, and whether chads are properly removed from the punch holes.

[0029] Camera 102a detects when and how much of a page is on the platen, then an image is captured and the corners are located using a line detection algorithm.

[0030] The differences in position and orientation between the desired page placement (model) and the placed sheet (image) are calculated. If the differences are within a given tolerance, the Inspector notifies the main process that the page is good and the page continues. On the other hand, if the differences are not within the tolerance, the Inspector notifies the main process that the page placement is improper and provides the captured image and the measured position and orientation. Similarly, after printing and punching, another image is captured and a circle detection algorithm is used to verify that the punch holes are properly formed in the correct positions and that chads are removed.

[0031] If the main process determines that corrective action is required, it first removes the defective page on the platen by turning off the powder ring page detector (to prevent wasteful powder deposition on the defective page) and transporting the defective page to the powder ring system. Once the page has passed through the powder ring system and is deposited on the rear conveyor, the reverse flow gate reverses the direction of the rear conveyor so that the sheet is deposited in the rejected page bin 110. The conveyor direction is then reversed and the page counter is reset to reprint the current page. At this time, the printing process of the page is restarted. If the main process detects consecutive misplacement of pages, it may adjust the motor control parameters to correct the placement position (e.g., move slightly in the Y direction) or notify the operator to make corrections.

[0032] 2) Rear conveyor monitor C2 102b The monitor analyzes the pages from the powder ring system and can determine if the image is placed within punch hole tolerances, if there are printing issues (e.g., low or empty ink, low powder deposition level, missing print swaths, clogged inkjet nozzles, inkjet head leaks, etc.), if the punch holes are in the proper location, if the punch holes are shaped properly, and if the punch hole chads have been properly removed. The monitor also analyzes the powder deposition to determine the powder deposition level, the distribution of the powder deposition (i.e., uniformity of coverage), powder agglomeration, and the residual powder level (i.e., the amount of powder remaining in the unprinted areas of the page). Additionally, the monitor can detect pages caught on the conveyor and the orientation of the page on the belt.

[0033] The monitor uses camera C2 102b to capture an image of the page after powdering is complete. First, the image is filtered to perform spatial (e.g., perspective) and illumination correction, and the image is scaled to the same resolution. Perspective correction, as well as any computational lens correction known in the art, either conventional or future, may be used to pre-process and / or tune the optical system for the classification algorithm. The captured image (i.e., the actual page) is then compared to the model image. Several methods are used to determine whether the page is acceptable. First, the captured image is analyzed to determine whether it deviates significantly from the model image. The root mean square difference indicates the overall level of deviation. Line detection or neural networks are used to detect specific artifacts to determine whether the image is properly formed. The problem can also be traced back to the print head or nozzles from the anomaly location.

[0034] The contrast between printed and unprinted areas and area detection (and an indication of the uniformity of the powder distribution) are used to calculate the quality of the powder deposition, and circle detection is used to verify the distance between the image and the printed image.

[0035] If the printed page is determined to be defective (i.e., outside of tolerance limits), it is removed from the print pipeline by redirecting the defective page to the rear conveyor for collection in the rejected page bin 110. Once the defective page has been removed, the main process performs corrective actions such as cleaning and wiping the heads, adding ink or powder, and modifying powder ring parameters to adjust powder and residual powder levels. Once complete, the page counter is reset so that the current page is reprinted and the page printing process is restarted.

[0036] 3) Stacker Monitor C3 102c The monitor detects whether the page is properly stacked on all four registration pins. When the page enters the stacker, the stacker pushes the page to the set position of the registration pins in the camera's field of view without touching the already stacked pages. Then the monitor process receives a signal (from the main process) and captures the image of the stacked page.

[0037] It then measures the visibility of each pin, detects whether any corners extend beyond the stacking area, and uses image analysis to determine if the page is properly aligned on all four registration pins, and if not, which pins are not properly aligned. Because there was previously no automated way to remove incorrectly stacked sheets, the print job is paused for an operator to manually remove or re-stack the sheets. Once the sheets are removed, the page counter is reset and the print job resumes.

[0038] The main process then determines what corrective action is required based on the classification and auxiliary data provided by the monitor: if the layer is classified as a good page (i.e., meets the tolerance criteria), it proceeds to the next step in the printing process, and one or more repair steps are performed, if necessary.

[0039] On the other hand, if the page is classified as a bad page, first all bad pages in the printing pipeline are removed. This can be done by pausing the machine and asking the operator to manually remove the sheet, or it can be done automatically by transporting the page to the rear conveyor and reversing the direction of the rear conveyor via a reverse flow gate (where the conveyor direction then returns to the direction of page flow) so that the page is deposited in the rejected page bin 110. The above operations are repeated until all bad pages in the printing pipeline have been removed. After the pages have been removed, an appropriate combination of the following remedial steps are performed:

[0040] Resolves low ink conditions by pausing the print job and refilling the ink. The print job will resume once ink levels are sufficient.

[0041] Inkjet problems can be resolved by moving the print head over the ink cuspidor, cleaning the ink head, and then moving the print head over the wiper blade to wipe the abnormal print head.

[0042] A powder shortage condition is resolved by initiating a powder refill operation.

[0043] Other powder problems, such as unacceptable powder build-up, are solved by changing powder ring parameters such as vacuum level, powder trough opening, air knife pressure, belt speed, etc.

[0044] If possible, an automated test is performed to test if the problem has been corrected. This may involve reading sensor values ​​(e.g. powder level) or printing a diagnostic image (a test page is then transported to a reject page bin). If an autonomous correction cannot be applied or the correction process does not resolve the problem, the print job is paused and an operator is notified to manually correct the problem. Finally, when the print job is resumed, the page counter is reset and printing resumes where it left off.

[0045] In some embodiments, the system may implement certain practices to aid in monitoring and classification. Table II provides non-limiting examples of how such practices may be deployed.

[0046] [Table II] TIFF0007601419000003.tif212154TIFF0007601419000004.tif233153TIFF0007601419 000005.tif93153TIFF0007601419000006.tif230153TIFF0007601419000007.tif93153

[0047] 2-4 are detailed schematic diagrams of an embodiment of the apparatus and monitoring system. As can be seen in Fig. 2, the print platen camera Cl 102a is shown at the top of the figure, the camera illumination 203 is also above the platen 209, and a sheet 205 is placed on the platen.

[0048] Figure 3 shows the rear conveyor camera C2 102b. The rear conveyor camera light 303 is located near the camera.

[0049] FIG. 4 shows stacker camera C3 102c, four alignment pins 407a, b, c, d, and a misaligned page 409.

[0050] Figure 5 is a flow chart of a typical corrective printing process performed using the print platen camera Cl 102a. At print start 502, the main process notifies the camera unit of an arriving page 504. The camera sends an image for analysis to the main processor (or image processor) where image classification is performed 506. The main processor then receives the page classification 508 and determines whether the page is good or whether corrective action is required (at least one required page parameter is outside of acceptable range) 510. If the page is good, the print cycle is terminated 516. If the page is bad, corrective action is performed 512. In all cases, data from each step is recorded 514.

[0051] FIG. 6 is a flow chart of the classification process after powder deposition. When a page is printed and on the rear conveyor, a page in progress message is received from the main process 602. Using camera C2 102b, the system waits for the page to enter the camera view 604. If the page is not detected within a timeout period, a "page missing" message is sent 624. If the page is detected, the camera captures the page image 608. Corrections are made for perspective and lighting conditions 609 (as mentioned above, perspective correction, as well as any computational lens correction known in the art, conventional or future, may be used to pre-process and / or calibrate the optical system for the classification algorithm). The corrected scanned image is then compared 610 to one or more stored model images. The root mean square (RMS) difference between the scanned image and the model is calculated 612, and a determination is made about the RMS error between the scanned image and the model 614. If the powder deposited page is within tolerance, a good page classification is sent 616. If the RMS error is too large, a specific bad page classification is determined 618. The bad page classification and any associated classification data are sent 620. In either case, the classification results are sent 622 to the main process.

[0052] The use of root mean square (RMS) difference is one technique that can be used and is shown in the figure as an example. Any individual monitoring and associated classification system is not limited to RMS and may use any image analysis technique, such as neural network classification and other forms of artificial intelligence classification. Indeed, the analysis of tolerances, errors and parameters, whether within tolerance or out of tolerance, can be accomplished using any classifier or image analysis system known in the art, either conventionally or in the future. The purpose of classification is to determine whether the page is within the tolerance range of the individual parameters that are generally applicable or specific to a particular build.

[0053] FIG. 7 is a flow chart of the corrective process. When a bad page is detected at any of the monitoring locations, the corrective process is started 702. In case of a bad page classification 704, the sheet is removed from the pipeline 712 and the page counter is reset 714. In case of a good page classification, these two steps are skipped. Then, it is determined 706 whether further corrective action is required. If no further correction is required, printing continues 708 and the corrective process cycle ends 710. On the other hand, if further corrective action is required, it is determined 716 whether an automatic correction is possible. If an automatic correction is possible, it is performed 718. If an automatic correction is not possible, the job is paused and an operator is notified 720. Then, it is waited for the operator to indicate that the problem has been solved 722. After the automatic or manual corrective action, a test is performed 724 to check whether the problem has really been solved. Once the test sheet has been removed from the pipeline 728, it is determined 728 whether the problem has really been solved. In the affirmative, it is determined 706 whether further (or other) corrective action is required. If corrective action is not required, printing continues 708 and the corrective action cycle ends 710. If corrective action is required, the issue of automatic or manual correction is again determined 716.

[0054] Although the above description uses the example of a sheet like substrate, the inventive principles described herein apply equally to web or roll-based supplies of substrate material.

[0055] The above description and drawings are intended to facilitate understanding of the present invention. It is obvious to those skilled in the art that various modifications and variations are possible without departing from the spirit of the present invention. These modifications and variations are within the scope of the present invention.

Claims

1. 1. A camera surveillance subsystem for a Composite Based Additive Manufacturing (CBAM) process, comprising: a print platen monitor configured to observe page placement of a sheet on a CBAM print platen, the print platen monitor comprising a first camera system for determining at least one of: a) whether there is a single page on the print platen; b) whether the position and angle of the single page are within an acceptable range; c) whether registration holes of the single page have proper shapes and locations; and d) whether chads have been properly removed from punched registration holes; a rear conveyor monitor configured to analyze a single page from the CBAM powder ring system to determine at least one of: a) whether a printed image is positioned within a tolerance for punch holes in the single page; b) whether the printed image has printing defects; c) whether punch holes are properly located; d) whether the shape of the punch holes is proper; e) whether punch hole chads have been properly removed; and f) whether powder deposition on the single page is within an acceptable range; and a stacker monitor including a third camera disposed on the CBAM stacker and configured to determine whether the single page is properly stacked on the registration pins and, if not, to identify which pins are missing; acquiring an image using the print platen monitor, the rear conveyor monitor, and the stacker monitor, perspective correcting the image, and automatically determining whether the single page is within an acceptable range using a root mean square (RMS) difference between the perspective corrected image and an ideal image; If the single page is determined to be unacceptable, (1) determining whether automatic correction is possible for the CBAM process, and if automatic correction is not possible, pausing the print job, notifying an operator, and waiting for an instruction from the operator that the problem has been resolved; (2) if an automatic correction is possible, performing the automatic correction, running a test to determine if the problem was corrected, removing the single page, and resuming printing; and a control system for performing corrective actions including 1. A camera monitoring subsystem for a composite-based additive manufacturing (CBAM) process, comprising:

2. The camera surveillance system of claim 1 , wherein the rear conveyor monitor determines powder pile level, powder pile distribution, powder agglomeration, and residual powder level.

3. The camera surveillance system of claim 1 , wherein the rear conveyor monitor detects the single page caught on a conveyor and the orientation of the single page on the conveyor belt.

Images