3D printer control method and system thereof

Abstract

A 3D printer control method and a system thereof are provided. The 3D printer control method according to some embodiments may comprise the steps of: acquiring design data for each layer for a product; receiving, in real time, a captured image for a current layer of the product output from a stacking-type 3D printer; performing an inspection for an output error of the current layer by comparing the captured image with the design data of the current layer acquired from the design data for each layer; and stopping an output of the 3D printer on the basis of a result of the inspection showing that the output error exists in the current layer. According to the method, problems such as unnecessary material waste and process delays caused by delay in determining a defective product can be minimized.

Description

3D printer control method and system thereof

[0001] The present invention relates to a 3D printer control method and system thereof, and more particularly, to a method and system thereof for monitoring and controlling the output process of a 3D printer in real time.

[0002]

[0003] Additive 3D printers are printers that output products by stacking very thin layers one after another. The manufacturing process based on these 3D printers can be divided into three stages: the modeling process, which designs the product; the printing process, which creates the product by layering materials (e.g., filament, resin, etc.); and the post-processing process, which involves drying, polishing, or dyeing the printed product.

[0004] Meanwhile, when mass-producing products using 3D printers, 3D scanner-based quality assessments are typically performed only on completed products. This means that even if errors occur during the printing process, defects are determined after the product is completed. This delay in identifying defects leads to various inefficiencies, such as unnecessary material waste and process delays. Therefore, a technology capable of detecting printing errors in real time during the printing process is needed.

[0005]

[0006] A technical problem to be solved through some embodiments of the present disclosure is to provide a method and system capable of detecting output errors in real time during the output process of a 3D printer.

[0007] Another technical problem to be solved through some embodiments of the present disclosure is to provide a method and system capable of accurately detecting various output errors of a 3D printer.

[0008] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.

[0009]

[0010] In order to solve the above technical problem, a method for controlling a 3D printer according to some embodiments of the present disclosure may include a step of obtaining layer-by-layer design data for a product, a step of receiving a photographed image of a current layer of the product output from a stacking-type 3D printer in real time, a step of performing an inspection for an output error of the current layer by comparing the design data of the current layer obtained from the layer-by-layer design data with the photographed image, and a step of stopping the output of the 3D printer based on a result of the inspection that an output error exists in the current layer.

[0011] In some embodiments, the step of checking for output errors of the current layer may further include the step of comparing design data of the current layer with a depth map of the current layer.

[0012] In some embodiments, the captured image includes a first captured image received at a first point in time and a second captured image received at a second point in time, and the step of performing a check for an output error of the current layer includes a step of comparing each of the first captured image and the second captured image with design data of the current layer, and an interval between the first point in time and the second point in time can be determined based on a complexity of the current layer.

[0013] In some embodiments, the step of performing a check for an output error of the current layer may include the steps of calculating a complexity of the current layer, adjusting a tolerance to be applied to the current layer based on the complexity, and performing a check for the output error based on the adjusted tolerance.

[0014] In some embodiments, the method may further include a step of calculating a difference between the current layer and the previous layer based on the layer-by-layer design data, and a step of performing an output error check for layer missing based on a determination that the calculated difference is greater than or equal to a reference value.

[0015] In some embodiments, the method may further include a step of calculating a difference between the current layer and the previous layer based on the layer-by-layer design data, and a step of performing an output error check regarding a layer shift based on a determination that the calculated difference is less than a reference value.

[0016] In order to solve the above-described technical problem, a 3D printer control system according to some embodiments of the present disclosure includes one or more processors and a memory storing a computer program executed by the one or more processors, wherein the computer program may include instructions for an operation of obtaining layer-by-layer design data for a product, an operation of receiving a photographed image of a current layer of the product output from a stacking-type 3D printer in real time, an operation of performing an inspection for an output error of the current layer by comparing the design data of the current layer obtained from the layer-by-layer design data with the photographed image, and an operation of stopping the output of the 3D printer based on a result of the inspection that an output error exists in the current layer.

[0017] In order to solve the above-described technical problem, a computer program according to some embodiments of the present disclosure may be stored in a computer-readable recording medium in combination with a computing device to execute the steps of: obtaining layer-by-layer design data for a product; receiving in real time a photographed image of a current layer of the product output from a stacking-type 3D printer; comparing the design data of the current layer obtained from the layer-by-layer design data with the photographed image to perform a check for an output error of the current layer; and stopping the output of the 3D printer based on a result of the check that an output error exists in the current layer.

[0018]

[0019] According to some embodiments of the present disclosure, output errors in a 3D printer can be detected in real time by comparing the captured image of the current layer with design data. Furthermore, in response to the detection of an output error, the output (operation) of the 3D printer can be immediately halted. In this case, problems such as unnecessary material waste and process delays caused by delays in determining defective products can be minimized.

[0020] Additionally, output error checking can be performed in a differential manner depending on the complexity of the layer. In this case, output error checking can be performed more intensively in complex layers, resulting in improved performance efficiency and accuracy.

[0021] Additionally, output error checking for layer shifts or missing layers can be selectively performed, taking into account differences between the current layer and the previous layer. In this case, the efficiency of output error checking can be improved, as checking for both output errors is no longer required across all layers. Furthermore, by performing checks for various output errors, output errors present in the product can be detected more thoroughly.

[0022] Furthermore, by training a deep learning model using multiple tasks related to depth map prediction, the depth map prediction performance of the deep learning model can be significantly improved. Furthermore, by utilizing this deep learning model, accurate and precise depth maps can be obtained even without a depth camera module.

[0023] The effects according to the technical idea of ​​the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0024]

[0025] FIG. 1 and FIG. 2 are exemplary schematic diagrams for schematically explaining a 3D printer control system according to some embodiments of the present disclosure.

[0026] FIG. 3 is an exemplary flowchart illustrating a 3D printer control method according to some embodiments of the present disclosure.

[0027] FIG. 4 is an exemplary diagram illustrating a depth map-based output error checking method according to some embodiments of the present disclosure.

[0028] FIGS. 5 and 6 illustrate monitoring screens of a control system that may be referenced in some embodiments of the present disclosure.

[0029] FIG. 7 is an exemplary flowchart illustrating a 3D printer control method according to some other embodiments of the present disclosure.

[0030] FIG. 8 is an exemplary diagram illustrating a layer complexity calculation method according to some embodiments of the present disclosure.

[0031] FIG. 9 is an exemplary diagram illustrating a layer complexity calculation method according to some other embodiments of the present disclosure.

[0032] FIG. 10 is an exemplary flowchart illustrating a 3D printer control method according to some further embodiments of the present disclosure.

[0033] Figures 11 and 12 are exemplary drawings to further explain the detailed process of the output error inspection step considering the layer difference illustrated in Figure 10.

[0034] FIG. 13 and FIG. 14 are exemplary drawings for explaining the structure and learning method of a deep learning model (i.e., a deep learning model for generating a depth map) according to some embodiments of the present disclosure.

[0035] FIG. 15 and FIG. 16 are exemplary diagrams for explaining a data augmentation method for training a deep learning model according to some embodiments of the present disclosure.

[0036] FIG. 17 illustrates an exemplary computing device that may implement a control system according to some embodiments of the present disclosure.

[0037]

[0038] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the attached drawings. However, the technical idea of ​​the present disclosure is not limited to the following embodiments and may be implemented in various different forms. The following embodiments are provided only to complete the technical idea of ​​the present disclosure and to fully inform those skilled in the art of the present disclosure of the scope of the present disclosure, and the technical idea of ​​the present disclosure is defined only by the scope of the claims.

[0039] When assigning reference numerals to components in each drawing, it should be noted that identical components are assigned the same numerals whenever possible, even if they appear on different drawings. Furthermore, when describing the present disclosure, if a detailed description of a related known configuration or function is deemed likely to obscure the gist of the present disclosure, such detailed description will be omitted.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in the same sense as commonly understood by those of ordinary skill in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terminology used herein is for the purpose of describing embodiments and is not intended to limit the disclosure. In this specification, singular forms also include plural forms, unless specifically stated otherwise.

[0041] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When a component is described as being "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be "connected," "coupled," or "connected" between each component.

[0042] As used herein, the terms “comprises” and / or “comprising” do not exclude the presence or addition of one or more other components, steps, operations and / or elements.

[0043] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0044] FIG. 1 is an exemplary schematic diagram for schematically explaining a 3D printer control system according to some embodiments of the present disclosure.

[0045] As illustrated in FIG. 1, the 3D printer control system according to the embodiments is a system capable of efficiently manufacturing a desired product (14) through appropriate control of a 3D printer (11). The 3D printer control system may be configured to include a control system (10), a 3D printer (11), and a camera module (12). However, in some cases, the control system (10) and the camera module (12) may be referred to as a '3D printer control system', or the control system (10) may be referred to alone as a '3D printer control system'.

[0046] The control system (10) is a computing device / system that performs real-time monitoring and control of a 3D printer (11). For example, the control system (10) can receive, in real time, a photographed image (video) of a product (14) output from the 3D printer (11) from the camera module (12) and analyze these photographed images to detect an output error of the 3D printer (11) (i.e., an output error existing in the product (14)). Since the 3D printer (11) outputs the product (14) in a layered manner, the control system (10) can receive a photographed image of the current layer in real time and detect an output error existing in the current layer in real time. In addition, the control system (10) can immediately stop the output (i.e., operation) of the 3D printer (11) in response to the detection of an output error. By doing so, problems such as unnecessary material exhaustion and process delays can be minimized.

[0047] The specific monitoring and control method of the control system (10) will be described in detail later with reference to the drawings below FIG. 3.

[0048] The control system (10) described above may be implemented with at least one computing device. For example, all functions of the control system (10) may be implemented on a single computing device, or a first function of the control system (10) may be implemented on a first computing device and a second function may be implemented on a second computing device. Alternatively, specific functions of the control system (10) may be implemented on multiple computing devices.

[0049] A computing device may include any type of device having computing (processing) capabilities, and for an example of such a device, see FIG. 17.

[0050] Note that a computing device is a collection of interacting components (e.g., memory, processor, etc.), and thus may sometimes be referred to as a "computing system." Of course, the term "computing system" can also encompass the concept of a collection of interacting computing devices.

[0051] Next, the camera module (12) is a module that photographs a product (14) output from a 3D printer (11) in real time. For example, the camera module (12) can be placed (installed) above the product (14) to photograph the layers that make up the product (14) directly below.

[0052] Although FIG. 1 illustrates an example in which one camera module (12) is installed on the top of a 3D printer (11), the scope of the present disclosure is not limited thereto. For example, a camera module (not shown) for photographing layers of a product (14) from the side, a camera module (not shown) for photographing the product (14) from an angle other than directly downward, etc. may be further installed inside the 3D printer (11). In addition, in some cases, depth camera modules (not shown) capable of generating a depth map (i.e., depth image) for the product (14) from various angles may be further installed.

[0053] The camera module (12) can transmit the captured image (video) to the control system (10) in real time. By doing so, output errors of the 3D printer (11) can be detected and controlled in real time.

[0054] Next, the 3D printer (11) is a printing device having a manufacturing capability for a 3D product (14). As illustrated, the 3D printer (11) may be a printing device that manufactures a product (14) using a layered method. For example, the 3D printer (11) may be a printing device that operates using an SLA (Stereo Lithography Apparatus) method or an FDM (Fused Deposition Modeling) method, but the scope of the present disclosure is not limited thereto.

[0055] For reference, Fig. 1 illustrates an example in which a 3D printer (11) manufactures a product (14) by stacking layers on a bed (13).

[0056] In some embodiments, as illustrated in FIG. 2, the manufacturing system (10) and the 3D printer (11) may be manufactured in the form of an integrated mass production chamber. For example, the integrated mass production chamber may be manufactured in a form in which the output process of the 3D printer (11) can be confirmed with the naked eye and on the screen (display) of the control system (10) (e.g., whether an output error exists in the current layer (22) of the product can be confirmed with the naked eye and on the screen), but the scope of the present disclosure is not limited thereto.

[0057] So far, a 3D printer control system according to some embodiments of the present disclosure has been schematically described with reference to FIGS. 1 and 2. Hereinafter, various methods that can be performed in the control system (10) will be described in detail.

[0058] Hereinafter, for ease of understanding, the description will proceed under the assumption that all steps / operations of the methods to be described below are performed in the control system (10) of the environment illustrated in FIG. 1 and / or FIG. 2. Accordingly, if the subject of a specific step / operation is omitted, it can be understood that it is performed in the control system (10). However, in an actual environment, some steps / operations of the methods to be described below may be performed in other computing devices.

[0059] Additionally, in the following, reference numbers may be omitted when not directly referring to the drawings, and reference numbers for the same components may be changed depending on the embodiment for the clarity of the present disclosure.

[0060] Figure 3 is an exemplary flowchart illustrating a 3D printer control method according to some embodiments of the present disclosure. However, this is merely a preferred embodiment for achieving the purpose of the present disclosure, and it is understood that some steps may be added or deleted as needed.

[0061] As illustrated in FIG. 3, the present embodiments may begin at step S31 of acquiring layer-by-layer design data for a product.

[0062] Layer-by-layer design data is data that can be distinguished by layer and can include various data / information that can be obtained during the product design (modeling) stage (process).

[0063] For example, the design data for each layer may include, but is not limited to, a G-code file obtained through a slicer program, data analyzed for each layer of 3D design data of a product (e.g., design images (blueprints), design dimensions, etc.) (e.g., design images (blueprints) for each layer, design dimension data for each layer, etc.). Those skilled in the art will likely already be familiar with 'G-code', so an explanation thereof will be omitted.

[0064] In step S32, a captured image of the current layer of the product output from the 3D printer can be received in real time. For example, the control system (10) can receive an image of the current layer of the product captured directly downward from a camera module (12) positioned above the product. For examples of images captured in this way, refer to FIGS. 5 and 6, etc.

[0065] In step S33, the design data of the current layer may be compared with the captured image. In other words, output error checking may be performed by comparing the design data of the current layer obtained from the layer-specific design data with the captured image. For example, the control system (10) may compare the design image of the current layer with the captured image, or may compare the design dimensions of the current layer with the manufacturing dimensions obtained from the captured image. The specific comparison method may be any method.

[0066] Meanwhile, the timing and / or cycle for performing output error checking can be set in various ways. For example, the control system (10) may perform output error checking for the current layer (see S33) at the time when the output of the current layer is completed, or may repeatedly perform output error checking (see S33) according to a preset cycle while the current layer is being output. In this case, the preset cycle may be dynamically determined based on the complexity of the current layer; for this, refer to the descriptions of FIGS. 7 to 9.

[0067] In some embodiments, output error checking for the current layer may be performed further based on the depth map for the current layer. For example, as illustrated in FIG. 4, assume that a depth map (41, where a lighter color indicates a shallower depth (closer distance)) corresponding to the current layer image photographed directly below has been acquired. This depth map (41) may be generated via a deep learning model (see FIG. 13) to be described later or via a depth camera module installed on top of the product. In this case, the control system (10) may further compare the depth map (41) with design data (e.g., design dimensions for depth) of the current layer to check for output errors of the current layer (e.g., comparing the depth value of the product area (42) in the depth map (41) with the corresponding design dimensions (see 43) to determine whether the manufacturing error exceeds the tolerance). By doing so, the check for output errors can be performed more thoroughly.

[0068] In some cases, depth map-based output error checking may be performed based on a judgment that the complexity of the current layer is higher than a reference value. That is, the control system (10) may additionally perform depth map-based output error checking only for complex layers. In this case, since depth map-based output error checking is selectively performed only for layers that require it, the efficiency of performing output error checking can be further improved.

[0069] In step S34, based on the comparison result, it can be determined whether an output error of the 3D printer exists in the current layer. For example, if the manufacturing error according to the comparison result exceeds a preset tolerance, the control system (10) can determine that an output error exists (has occurred) in the current layer. At this time, the tolerance may be set to the same value for each layer or may be set differently. For example, the tolerance may be set (determined) to different values ​​based on the complexity of the layer; for this, refer to the description of FIGS. 7 to 9. In some cases, the tolerance may be set (determined) to a value inversely proportional to the height of the layer (e.g., the smaller the layer height is set, the smaller the tolerance may be determined to ensure the precision of the printed product). Depending on the determination result of this step, step S35 or step S36 may be performed.

[0070] In step S35, the output (operation) of the 3D printer may be stopped. For example, the control system (10) may immediately transmit a control command to the 3D printer (11) to stop output based on a judgment (i.e., inspection result) that an output error exists in the current layer. By doing so, problems such as unnecessary material exhaustion and process delays can be minimized.

[0071] In addition, as illustrated in FIG. 5, the control system (10) may display information regarding output errors on the monitoring screen (52). For example, the control system (10) may display information regarding the area (51) in which output errors occur in the current layer, the type of output errors, the cause of occurrence, and measures to be taken, on the monitoring screen (52).

[0072] For reference, Fig. 6 illustrates a monitoring screen (61) when no output error occurs.

[0073] This is explained again with reference to Figure 3.

[0074] In step S36, it can be determined whether the current layer is the last layer of the product. If the current layer is not the last layer, steps S32 to S34 described above can be repeated for the next layer.

[0075] Hereinafter, a 3D printer control method according to some embodiments of the present disclosure has been described with reference to FIGS. 3 through 6. As described above, output errors in a 3D printer can be detected in real time by comparing a captured image of the current layer with design data. Furthermore, in response to the detection of an output error, the output (operation) of the 3D printer can be immediately halted. In this case, problems such as unnecessary material depletion and process delays caused by delays in determining defective products can be minimized.

[0076] Hereinafter, a 3D printer control method according to several other embodiments of the present disclosure will be described with reference to FIGS. 7 to 9. However, for the clarity of the present disclosure, descriptions of content overlapping with the previous embodiments will be omitted.

[0077] FIG. 7 is an exemplary flowchart illustrating a 3D printer control method according to several other embodiments of the present disclosure. However, this is merely a preferred embodiment for achieving the purpose of the present disclosure, and it is to be understood that some steps may be added or deleted as needed.

[0078] As illustrated in FIG. 7, the present embodiments relate to a method for performing output error checking while taking into account the layer complexity of a product. The present embodiments are described in detail below.

[0079] In step S71, layer-by-layer design data for the product can be obtained. For this, refer to the description of step S31 described above.

[0080] In step S72, a captured image of the current layer of the product being output from the 3D printer can be received in real time. For this, please refer to the description of step S32 described above.

[0081] In step S73, the complexity of the current layer can be calculated. However, the specific complexity calculation method may vary depending on the embodiment.

[0082] In some embodiments, the complexity of the current layer may be calculated based on the number of G-code commands. For example, as illustrated in FIG. 8, the control system (10) may count the number of G-code commands for the current layer in the G-code file (81) (e.g., 82 to 84 when the current layer is 'Layer 2', etc.) and calculate the complexity of the current layer as a value proportional to the number (i.e., the greater the number of G-code commands, the higher the complexity calculated as a value). In some cases, the control system (10) may assign different weights to different types of G-code commands and calculate the complexity of the current layer further based on these weights. In this case, the complexity of the current layer may be calculated more accurately.

[0083] In some other embodiments, the complexity of the current layer may be calculated based on the complexity of the design image. For example, the control system (10) may calculate the complexity of the current layer based on the degree of change in pixel values ​​of the design image of the current layer, the number of pixels with different values, etc. The complexity of the current layer may be calculated as a value proportional to the complexity of the design image.

[0084] In some other embodiments, the complexity of the current layer may be calculated based on the shape complexity of the current layer. For example, the control system (10) may calculate the complexity of the current layer as a value proportional to the number of vertices, edges, and / or faces associated with the current layer, the number of supports present in the current layer, the number of holes, etc.

[0085] In some other embodiments, the final complexity of the current layer may be calculated as a weighted sum of the complexity of the first part belonging to the product in the current layer and the complexity of the second part belonging to the support. At this time, the weight of the first part may be set to a higher value than the weight of the second part. For example, as shown in FIG. 9, assume that there are a first product (91) and a second product (92), and the current layer is 'layer 2'. In this case, the current layer complexity of the first product (91) may be calculated to be a lower value than that of the second product (92). This is because the current layer of the first product (91) includes more support parts (e.g., 93) than that of the second product (92) (compare 92 with 94, compare 93 with 95).

[0086] In some other embodiments, the complexity of the current layer may be calculated based on various combinations of the above-described embodiments. For example, the control system (10) may calculate the final complexity for the current layer by aggregating (e.g., weighted sum) the first complexity calculated according to the first embodiment (e.g., based on the number of G-code commands) and the second complexity calculated according to the second embodiment (e.g., based on shape complexity).

[0087] This is explained again with reference to Figure 7.

[0088] In step S74, output error checking for the current layer may be performed considering the complexity of the current layer. However, the specific output error checking method may vary depending on the embodiment.

[0089] In some embodiments, the execution cycle (interval) of the output error check may be determined based on the complexity of the current layer. For example, assume that while the current layer is being output, a first captured image is received at a first point in time, and a second captured image is received at a second point in time, which is after the first point in time. Furthermore, assume that the control system (10) performs the output error check by comparing the first captured image with the design data of the current layer, and then again comparing the second captured image with the design data of the current layer. In this case, the interval (i.e., the interval) between the first and second points in time may be determined based on the complexity of the current layer. For example, the higher the complexity of the current layer, the smaller the interval between the two points in time may be determined. By doing so, the output error check can be performed more thoroughly for complex layers.

[0090] In some other embodiments, the tolerance to be applied to the current layer may be adjusted based on the complexity of the current layer, and output error checking for the current layer may be performed based on the adjusted tolerance. For example, the control system (10) may adjust the tolerance to a smaller value as the complexity of the current layer increases. By doing so, a more stringent error checking standard may be applied to the complex layer. In some cases, the control system (10) may count the number of layers among all layers constituting the product whose complexity is greater than a threshold value and adjust the tolerance to a value inversely proportional to the number (e.g., the more complex layers there are, the more sophisticated the product is, so the tolerance is adjusted to a smaller value).

[0091] In some other embodiments, further depth-map-based output error checking may be performed based on a determination that the complexity of the current layer exceeds a threshold. For this, see the description of FIG. 4.

[0092] In some other embodiments, output error checking may be performed based on various combinations of the embodiments described above.

[0093] In steps S75 and S76, the output (operation) of the 3D printer may be stopped based on the determination that an output error exists in the current layer. For this, please refer to the description of steps S34 and S35 described above.

[0094] In step S77, it can be determined whether the current layer is the last layer of the product. For this, refer to the description of step S36 described above.

[0095] Hereinafter, a 3D printer control method according to several other embodiments of the present disclosure has been described with reference to FIGS. 7 through 9 . As described above, output error checking can be performed in a differential manner depending on the complexity of the layers. In this case, output error checking can be performed more intensively in complex layers, thereby improving both the efficiency and accuracy of output error checking.

[0096] Hereinafter, a 3D printer control method according to several other embodiments of the present disclosure will be described with reference to FIGS. 10 to 12. However, for the clarity of the present disclosure, descriptions of content overlapping with the previous embodiments will be omitted.

[0097] Figure 10 is an exemplary flowchart illustrating a 3D printer control method according to several further embodiments of the present disclosure. However, this is merely a preferred embodiment for achieving the purpose of the present disclosure, and it is to be understood that some steps may be added or deleted as needed.

[0098] As illustrated in FIG. 10, the present embodiments relate to a method for performing output error checking based on the difference between the current layer and the previous layer. The present embodiments are described in detail below.

[0099] In step S101, layer-by-layer design data for the product can be obtained. For this, refer to the description of step S31 described above.

[0100] In step S102, a captured image of the current layer of the product being output from the 3D printer can be received in real time. For this, please refer to the description of step S32 described above.

[0101] In step S103, the difference between the current layer and the previous layer may be calculated. For example, the control system (10) may calculate the difference between the two layers based on differences in shape, area, complexity, number of supports, average depth (e.g., depth measured from the side), etc. between the two layers. However, the scope of the present disclosure is not limited thereto.

[0102] In step S104, output error checking may be performed considering the difference between the two layers. However, the specific output error checking method may vary depending on the embodiment.

[0103] In some embodiments, an output error check regarding a layer shift may be performed based on a determination that the difference between two layers is less than a threshold. Those skilled in the art will likely be familiar with this 'layer shift error', so a description thereof will be omitted. For example, as illustrated in FIG. 11, the control system (10) may perform an output error check regarding a layer shift for two layers (113, 114) constituting a product (111) if the difference between the two layers (113, 114) is less than a threshold (assuming the current layer is layer 3 (113)). Specifically, the control system (10) obtains a side depth map (i.e., a depth map corresponding to a side-captured image) for the current layer (113) and the previous layer (114), and compares the depth value of the current layer (113) with the depth value of the previous layer (114) in the side depth map, thereby determining whether a layer shift error exists (e.g., determining whether the difference between the two depth values ​​corresponds to a difference in design dimensions). For reference, the side depth map may be received from a depth camera module or generated from a side-captured image via a deep learning model (see FIG. 13), and for this, refer to the descriptions of FIGS. 13 to 16.

[0104] In some other embodiments, an output error check for layer missing may be performed based on a judgment that the difference between two layers is greater than a threshold. Those skilled in the art will already be familiar with this 'layer missing error', so a description thereof will be omitted. For example, as illustrated in FIG. 11, if the difference between two layers (112, 113) constituting the product (111) is greater than a threshold (assuming the current layer is layer 4 (112)), the control system (10) may perform an output error check for layer missing for the layers preceding the current layer (112) (e.g., 113, 114). Specifically, as illustrated in FIG. 12, the control system (10) can receive an image taken from the side of previous layers (e.g., 113, 114) of the current layer (112) from the camera module (121), analyze the image, and count the number of previous layers (e.g., 113, 114). In addition, the control system (10) can determine whether a layer missing error exists by checking the number of previous layers (e.g., 113, 114) based on the design data for each layer (i.e., checking whether the number of layers matches the number of layers in the design data).

[0105] In some other embodiments, output error checking may be performed based on various combinations of the embodiments described above.

[0106] This is explained again with reference to Figure 10.

[0107] In steps S105 and S106, the output (operation) of the 3D printer may be stopped based on the determination that an output error exists. For this, please refer to the description of steps S34 and S35 described above.

[0108] In step S107, it can be determined whether the current layer is the last layer of the product. For this, refer to the description of step S36 described above.

[0109] Heretofore, a 3D printer control method according to several other embodiments of the present disclosure has been described with reference to FIGS. 10 to 12. As described above, output error checking for layer shift or layer disappearance can be selectively performed based on the difference between the current layer and the previous layer. In this case, since it is unnecessary to perform checks for both output errors in all layers, the efficiency of performing output error checks can be further improved. Furthermore, by performing checks for various output errors, output errors present in the product can be detected more thoroughly.

[0110] The various embodiments of the 3D printer control method described with reference to FIGS. 3 to 12 so far can be combined in various forms. For example, the control system (10) may perform output error checking by considering the complexity of the current layer, and at the same time, perform output error checking for layer shift and / or layer disappearance by considering the difference between the current layer and the previous layer.

[0111] Hereinafter, a method for generating a deep learning-based depth map according to some embodiments of the present disclosure will be described with reference to FIGS. 13 to 16.

[0112] As illustrated in FIG. 13, the depth map generation method to be described below relates to a method of generating a depth map (e.g., 137) from a two-dimensional image (e.g., 136) using a deep learning model. The training of such a deep learning model may be performed on another computing device, but for the sake of ease of understanding, the following explanation will continue by assuming that the training of the deep learning model is also performed by the control system (10).

[0113] First, the structure and operating principles of the deep learning model according to the embodiments will be described with reference to FIG. 13.

[0114] As illustrated in FIG. 13, a deep learning model according to embodiments may be configured to include an encoder (131), a decoder (132), and a plurality of predictors (133 to 135).

[0115] The encoder (131) is a neural network module that extracts features (e.g., feature maps) from an input image (136). The encoder (131) may be implemented based on a neural network such as a convolutional neural network (CNN) or a transformer, but the scope of the present disclosure is not limited thereto. The encoder (131) may be implemented in any manner. In some cases, a pretrained feature extractor (e.g., a pretrained VGG series neural network, etc.) may be used as the encoder (131). Since the encoder (131) is shared by multiple predictors (133 to 135), it may also be referred to as a 'shared encoder'.

[0116] Next, the decoder (132) is a neural network module that decodes features extracted from the encoder (131). The decoder (132) can increase the size of features (e.g., feature maps) by performing an up-sampling operation on the input features. The decoder (131) may be implemented based on, for example, a convolutional neural network, but the scope of the present disclosure is not limited thereto. Since the decoder (132) is also shared by a plurality of predictors (133 to 135), it may also be referred to as a “shared decoder.” In some cases, a deep learning model may be configured in a form in which the function of the decoder (132) is embedded in the predictors (133 to 135) (i.e., the decoder (132) may be omitted from the components of the deep learning model).

[0117] As illustrated, a skip connection that transmits features may exist between the encoder (131) and the decoder (132). For example, in a U-Net, each of the encoder (131) and the decoder (132) is composed of encoding blocks and decoding blocks at different levels, and a skip connection that functions as a feature transmission path may exist between the encoding blocks and decoding blocks at the same level. However, the scope of the present disclosure is not limited thereto.

[0118] The plurality of predictors (133 to 135) are neural network modules that perform different tasks. The plurality of predictors (133 to 135) can receive decoded features as input and perform corresponding tasks. Since the predictors (133 to 135) are layers that perform tasks, they may be referred to as "task-specific layers / modules" in some cases. The predictors (133 to 135) may be implemented based on a neural network such as a fully-connected layer, but the scope of the present disclosure is not limited thereto.

[0119] The first predictor (133) can predict and generate a depth map (137) for the input image (136) based on the decoded features. That is, the first predictor (133) can perform a depth map prediction task.

[0120] Next, the second predictor (134) can predict the class label of the input image (136) based on the decoded features. That is, the second predictor (134) can perform a class prediction task (i.e., a classification task).

[0121] Next, the third predictor (135) can predict and generate a segmentation map (138) for the input image (136) based on the decoded features. That is, the third predictor (135) can perform a semantic segmentation task that predicts classes on a pixel-by-pixel basis of the input image (136). Those skilled in the art will likely already be familiar with the deep learning task called 'semantic segmentation', and therefore, a description thereof will be omitted.

[0122] As illustrated, each of the predictors (133 to 135) may further utilize data from other predictors (e.g., prediction results, data within the predictor) to perform prediction (see arrows).

[0123] For reference, FIG. 13 illustrates an example in which all three predictors (133 to 135) are used in training a deep learning model, but in some cases, at least one of the second predictor (134) and the third predictor (135) may not be used in training a deep learning model.

[0124] Below, the learning method of the deep learning model will be described with reference to Fig. 14.

[0125] As illustrated in Fig. 14, a deep learning model can be trained by performing multi-tasks through multiple predictors (133 to 135). Specifically, each training sample constituting the training set can be composed of a two-dimensional image (141), a correct depth map (143, i.e., Ground Truth), a correct class label, and a correct segmentation map (145), and the control system (10) can train the deep learning model using a plurality of training samples configured in this manner.

[0126] More specifically, the control system (10) can input (feed) a two-dimensional image (141) into the encoder (131) and predict a depth map (142) for the input image (141) through the first predictor (133). Then, the control system (10) can calculate a first loss (L1) based on the difference between the predicted depth map (142) and the correct depth map (143). The method of calculating the first loss (L1) may be any method. In some cases, the control system (10) can calculate the first loss (L1) in a way that gives a higher weight to the difference (loss) of the internal area of ​​the object than to the external area. In this case, the deep learning model can be trained to perform prediction by focusing more on the internal area of ​​the object. This technical principle can also be applied to the third loss (L3).

[0127] In addition, the control system (10) can input (feed) a two-dimensional image (141) into the encoder (131) and predict the class label of the input image (141) through the second predictor (134). Then, the control system (10) can calculate the second loss (L2) based on the difference between the predicted class label and the correct class label. The second loss (L2) can be calculated based on, for example, a cross-entropy loss function, but the scope of the present disclosure is not limited thereto.

[0128] In addition, the control system (10) can input (feed) a two-dimensional image (141) to the encoder (131) and predict a segmentation map (144) for the input image (141) through a third predictor (135). Then, the control system (10) can calculate a third loss (L3) based on the difference between the predicted segmentation map (144) and the correct segmentation map (145). The third loss (L3) can be calculated based on, for example, a cross-entropy loss function, but the scope of the present disclosure is not limited thereto.

[0129] Next, the control system (10) can calculate a total loss by aggregating (e.g., weighting) the first loss (L1) to the third loss (L3) and update the parameters of the encoder (131), the decoder (132), and the predictors (133 to 135) based on the total loss. At this time, considering the importance of the task, the highest weight may be assigned to the first loss (L1) and the lowest weight may be assigned to the second loss (L2), but the scope of the present disclosure is limited thereto.

[0130] The above-described processes can be repeated for other training samples. By doing so, the deep learning model (e.g., the first predictor (133)) can be equipped with the ability to accurately generate a depth map corresponding to the input image (e.g., 141).

[0131] Meanwhile, in some embodiments, as illustrated, additional losses (e.g., L4 to L6) may be further utilized in training the deep learning model.

[0132] For example, the control system (10) can further calculate the fourth loss (L4) based on the difference between the boundary line extracted from the predicted depth map (142) and the boundary line extracted from the input image (141) (or the correct depth map (143)). The method for calculating the boundary line difference may be any method. In addition, the control system (10) can further update the parameters of the deep learning model based on the fourth loss (L4). In this case, the deep learning model can be trained to perform prediction by focusing more on the vicinity of the boundary line of the object.

[0133] As another example, the control system (10) can further calculate a fifth loss (L5) based on the difference between the boundary extracted from the predicted segmentation map (144) and the boundary extracted from the input image (141) (or the correct segmentation map (145)). Furthermore, the control system (10) can update the parameters of the deep learning model based on the fifth loss (L5). In this case, the deep learning model can be trained to perform predictions with greater focus on the vicinity of the boundary of the object.

[0134] As another example, the control system (10) can further calculate a sixth loss (L6) based on the difference between the boundary extracted from the predicted depth map (142) and the boundary extracted from the predicted segmentation map (144). Furthermore, the control system (10) can update the parameters of the deep learning model based on the sixth loss (L6). In this case, the deep learning model can be trained to perform predictions with greater focus on the vicinity of the object's boundary.

[0135] As another example, the control system (10) may calculate losses based on various combinations of the examples described above. For example, the control system (10) may further calculate fourth losses (L4) to sixth losses (L6) and update the parameters of the deep learning model based on these losses (L4 to L6).

[0136] Once the training of the deep learning model is completed, the control system (10) can generate a depth map for the captured image using the trained deep learning model. For example, the control system (10) can input a first image, which is taken directly below the current layer of the product, into the trained deep learning model to generate a first depth map, and can also input a second image, which is taken from the side of multiple layers of the product, into the trained deep learning model to generate a second depth map.

[0137] Meanwhile, in some embodiments, the control system (10) may generate new training samples using data augmentation techniques and further utilize these to train a deep learning model. By doing so, the performance of the deep learning model can be further improved. These embodiments will be further described with reference to FIGS. 15 and 16 below.

[0138] FIG. 15 and FIG. 16 are exemplary drawings for explaining a data augmentation method according to some embodiments of the present disclosure.

[0139] As illustrated in FIG. 15, the control system (10) can extract an object region (152) from an original image (151) to generate a new image (153). Here, the original image (151) refers to an original two-dimensional image existing in the training set, and the new image (153) can refer to an object image from which a non-object region (i.e., a background region) has been removed, for example.

[0140] Next, the control system (10) can extract the object area (155) from the correct depth map (154) of the original image (151) and generate a new correct depth map (156) for the new image (153). At this time, the control system (10) can generate the new correct depth map (156) by replacing the depth value corresponding to the non-object area in the correct depth map (154) with a constant.

[0141] By training a deep learning model using the new image (153) and the new correct depth map (156) as above, the deep learning model can be trained to predict depth values ​​by focusing more on the internal area of ​​the object.

[0142] In some cases, the control system (10) may select training samples (e.g., samples composed of a two-dimensional image and a correct depth map) in which the prediction loss (e.g., prediction loss for an object region) of a predictor (e.g., 133, 134, 135) in the training set is higher than a reference value (or the prediction accuracy is lower than a reference value), and may generate new training samples using the selected training samples.

[0143] Additionally, in some cases, the control system (10) may also determine the number of new training samples to be generated based on the prediction loss (e.g., prediction loss for the object region) of the predictor (e.g., 133, 134, 135) (e.g., generating more new training samples for training samples with high prediction loss).

[0144] Additionally, in some cases, the control system (10) may further generate a second new image in a manner that changes the color, shape and / or position of the object area (e.g., 152) in the first new image (e.g., 153).

[0145] Hereinafter, another example of a data augmentation method will be described with reference to FIG. 16.

[0146] As illustrated in Fig. 16, the control system (10) can generate a new image (163) by extracting only a portion (163) of the object area (152) of the original image (161). Here, the portion (163) may mean an area within the object area (162) where the prediction loss of the first predictor (i.e., the difference between the predicted depth value and the correct depth value) is greater than a reference value.

[0147] Next, the control system (10) can extract a portion (167) within the object area (166) from the correct depth map (165) of the original image (161) to generate a new correct depth map (168) for the new image (164). At this time, the control system (10) can generate the new correct depth map (168) by replacing the depth value corresponding to the non-object area in the correct depth map (165) with a constant.

[0148] By training a deep learning model using the new image (164) and the new correct answer depth map (168) as above, the deep learning model can be trained to more accurately predict the depth value of the internal area of ​​the object.

[0149] Hereinafter, a method for generating a deep learning depth map according to some embodiments of the present disclosure has been described with reference to FIGS. 13 to 16. As described above, by training a deep learning model using multiple tasks associated with depth map prediction, the depth map prediction performance of the deep learning model can be significantly improved. Furthermore, by utilizing such a deep learning model, an accurate and precise depth map can be obtained even without a depth camera module.

[0150] Hereinafter, an exemplary computing device (170) capable of implementing a control system (10) according to some embodiments of the present disclosure will be described with reference to FIG. 17.

[0151] Figure 17 is an exemplary hardware configuration diagram showing a computing device (170).

[0152] As illustrated in FIG. 17, a computing device (170) may include one or more processors (171), a bus (173), a communication interface (174), a memory (172) for loading a computer program (176) executed by the processor (171), and a storage (175) for storing the computer program (176). However, only components related to the embodiment of the present disclosure are illustrated in FIG. 17. Therefore, a person skilled in the art to which the present disclosure pertains will appreciate that other general components may be included in addition to the components illustrated in FIG. 17. That is, the computing device (170) may further include various components in addition to the components illustrated in FIG. 17. In addition, in some cases, the computing device (170) may be configured in a form in which some of the components illustrated in FIG. 17 are omitted. Hereinafter, each component of the computing device (170) will be described.

[0153] The processor (171) can control the overall operation of each component of the computing device (170). The processor (171) can be configured to include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), or any other type of processor well known in the art of the present disclosure. In addition, the processor (171) can perform operations for at least one application or program for executing operations / methods according to embodiments of the present disclosure. The computing device (170) can include one or more processors.

[0154] Next, the memory (172) can store various data, commands, and / or information. The memory (172) can load a computer program (176) from the storage (175) to execute operations / methods according to embodiments of the present disclosure. The memory (172) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0155] Next, the bus (173) can provide a communication function between components of the computing device (170). The bus (173) can be implemented as various types of buses such as an address bus, a data bus, and a control bus.

[0156] Next, the communication interface (174) can support wired and wireless Internet communication of the computing device (170). Furthermore, the communication interface (174) can also support various communication methods other than Internet communication. To this end, the communication interface (174) can be configured to include a communication module well known in the technical field of the present disclosure.

[0157] Next, the storage (175) can non-temporarily store one or more computer programs (176). The storage (175) can be configured to include non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0158] Next, the computer program (176) may include instructions that, when loaded into the memory (172), cause the processor (171) to perform operations / methods according to various embodiments of the present disclosure. That is, the processor (171) may perform operations / methods according to various embodiments of the present disclosure by executing the loaded instructions.

[0159] For example, the computer program (176) may include instructions for performing an operation of acquiring layer-by-layer design data for a product, an operation of receiving a photographed image of a current layer of a product output from a stacking 3D printer in real time, an operation of performing an operation of checking for an output error of the current layer by comparing the design data of the current layer obtained from the layer-by-layer design data with the photographed image, and an operation of stopping the output of the 3D printer based on an inspection result indicating that an output error exists in the current layer.

[0160] As another example, the computer program (176) may include instructions to perform at least some of the steps / operations described with reference to FIGS. 1 through 16.

[0161] In the case illustrated, a control system (10) according to some embodiments of the present disclosure may be implemented via a computing device (170).

[0162] So far, with reference to FIG. 17, an exemplary computing device (170) capable of implementing a control system (10) according to some embodiments of the present disclosure has been described.

[0163] The technical idea of ​​the present disclosure described with reference to FIGS. 1 to 17 so far can be implemented as computer-readable code on a computer-readable recording medium. The computer-readable recording medium can be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-attached hard disk). A computer program recorded on the computer-readable recording medium can be transmitted to another computing device via a network such as the Internet, installed on the other computing device, and thus used on the other computing device.

[0164] Although all components constituting the embodiments of the present disclosure have been described as being combined or operated in combination as one, the technical concept of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the present disclosure, all components may be selectively combined and operated one or more times.

[0165] Although operations are depicted in the drawings in a specific order, this should not be construed as requiring the operations to be performed in the specific order depicted, or in any sequential order, or as requiring all depicted operations to be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various components in the embodiments described above should not be construed as requiring such separation, and it should be understood that the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products.

[0166] Although various embodiments of the present disclosure have been described with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure can be implemented in other specific forms without changing the technical concepts or essential features thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included within the scope of the technical ideas defined by the present disclosure.

Claims

1. A method performed by at least one computing device, Step of acquiring layer-by-layer design data for a product; A step of receiving a real-time photographed image of the current layer of the product output from a 3D printer using a stacking method; A step of performing an inspection for an output error of the current layer by comparing the design data of the current layer obtained from the layer-by-layer design data with the photographed image; and A method for controlling a 3D printer, comprising the step of stopping output of the 3D printer based on the result of the inspection that an output error exists in the current layer.

2. In paragraph 1, The step of performing a check for output errors of the current layer above is: Including a step of further comparing the design data of the current layer and the depth map of the current layer, A 3D printer control method, wherein the step of further comparing the design data of the current layer with the depth map of the current layer includes the step of comparing the design data of the current layer with the depth map of the current layer based on a determination that the complexity of the current layer is greater than a reference value.

3. In paragraph 1, The above-mentioned captured image includes a first captured image received at a first point in time and a second captured image received at a second point in time, The step of performing a check for output errors of the current layer above is: Comprising a step of comparing each of the first captured image and the second captured image with the design data of the current layer, The interval between the first point in time and the second point in time is determined based on the complexity of the current layer, and the complexity of the current layer is calculated based on the number of G-code commands for the current layer. The design data of the current layer includes a design image, and the complexity of the current layer is calculated based on the complexity of the design image. The above current layer includes a first part belonging to the product and a second part belonging to support, The complexity of the current layer is calculated through a weighted sum of the complexity of the first part and the complexity of the second part, A 3D printer control method, wherein the weight of the first part is set to a higher value than the weight of the second part.

4. In paragraph 1, The step of performing a check for output errors of the current layer above is: A step of calculating the complexity of the current layer; A step of adjusting the tolerance to be applied to the current layer based on the above complexity; and A 3D printer control method, comprising a step of performing an inspection for the output error based on the adjusted tolerance.

5. In paragraph 1, A step of calculating the difference between the current layer and the previous layer based on the design data for each layer; and Further comprising a step of performing an output error check regarding layer missing based on a judgment that the calculated difference is greater than or equal to a reference value; The step of performing output error check for the above layer missing is: A step of receiving an image taken from the side of the previous layers of the current layer including the immediately preceding layer; A step of analyzing the above image and counting the number of the previous layers; and A 3D printer control method, comprising a step of determining whether an output error regarding the missing layer exists by checking the number of the previous layers based on the layer-by-layer design data.

6. In paragraph 1, A step of calculating the difference between the current layer and the previous layer based on the design data for each layer; and A 3D printer control method further comprising a step of performing an output error check regarding layer shift based on a determination that the calculated difference is less than a reference value.

7. In paragraph 6, The step of performing output error check for the above layer shift is: A step of obtaining a side depth map for the current layer and the previous layer; and A 3D printer control method, comprising a step of determining whether an output error regarding the layer shift exists by comparing the depth value of the current layer with the depth value of the previous layer in the side depth map.

8. In paragraph 7, The step of obtaining the above lateral depth map is: A step of receiving side images for the current layer and the previous layer; and A step of generating the side depth map from the side image using a deep learning model, The above deep learning model: A shared encoder that extracts features from the input image; A shared decoder for decoding the above extracted features; A first predictor that receives the decoded feature as input and predicts a depth map for the input image; A second predictor that receives the decoded feature as input and predicts the class label of the input image; and A third predictor is included that receives the decoded feature as input and predicts a segmentation map for the input image, A 3D printer control method, wherein the shared encoder and the shared decoder are learned based on the prediction loss of the first predictor to the third predictor.

9. In paragraph 8, The above shared encoder and the above shared decoder, The first loss calculated from the boundary difference between the input image and the predicted depth map, The second loss is calculated from the difference between the boundary lines of the input image and the predicted segmentation map. A 3D printer control method, wherein learning is further based on a third loss calculated from the boundary difference between the predicted depth map and the predicted segmentation map.

10. In paragraph 9, The above deep learning model is, The original image and the correct depth map for the original image, A first image generated by extracting an object area from the original image and a first depth map generated by replacing depth values ​​corresponding to non-object areas in the correct depth map with constants, and It is learned using a second image generated by extracting a part of the object area of ​​the original image and a second depth map for the second image generated from the first depth map, A 3D printer control method, wherein the above-mentioned partial area is an area in which the difference between the predicted depth value obtained through the first predictor and the correct depth value within the object area is greater than a reference value.

Images