Simulation-based synthetic defect generation for visual quality inspection

Abstract

In a computer-implemented method for supporting vision-based inspection of manufactured parts on a production line, a defect library is created in simulation utilizing real images of manufactured part samples with defects, by digitization and parameterization of enclosed defect geometries of different defect types obtained from the real images. Synthetic defects are generated by randomizing parameters defining the enclosed defect geometries of the defect types in the defect library. A 3D superimposition of the synthetic defects is carried out at randomized positions on a CAD model of a target part to generate a number of defect model objects, each including a digital representation of the target part containing one or more of the synthetic defects. Simulated camera images are then rendered using the generated defect model objects and a digital twin of an inspection environment having one or more cameras, to create a dataset for training a machine learning model.

Description

SIMULATION-BASED SYNTHETIC DEFECT GENERATION FOR VISUALQUALITY INSPECTIONTECHNICAL FIELD

[0001] The present disclosure relates to computer vision systems for inspection of manufactured parts on a production line. Specifically, this disclosure relates to methods, systems and computer program products for vision-based quality inspection of manufactured parts to detect defects using a machine learning model trained on a simulation-based synthetic defect generation framework.BACKGROUND

[0002] Deep learning models for computer vision typically require a large amount of labelled data for training. In most computer vision tasks, such as self-driving cars and object detection, access to the real data is streamlined and quite easy. However, in visual quality inspection of manufactured parts, specifically for supervised methods, acquiring sufficient amount of labeled data to train a model is often difficult and expensive.

[0003] Defects are typically rare in manufacturing. To collect enough data to train a model (e.g., -1000 images of defective parts on a production line), the vision system may have to be in data collection mode for several months. Furthermore, even with enough training data, the generalization capabilities of the model may still rely on the specific parts and defects that it is trained on. For example, if there is a similar defect on a completely different part, there is high probability that the model might miss the defect, and similarly, the false-positive rate will be increased due to overfitting on that one part. In the case of a completely new defect which hasn’t been part of the training dataset, the model might fail and may not be able to provide any detection.

[0004] Moreover, the task of labeling is often a burden to the operators on the shop floor. It is desirable for manufacturers such as OEMs and their suppliers to not incur any additional effort to the manufacturing workflow. Also, labeling can be subjective and often lacks consistency between different operators even for the same defects, creating a variability problem in the labels.

[0005] In machine learning, dataset size can be increased by applying data augmentation to the available training data. Data augmentation can help reduce overfitting of the model during training. Most common data augmentation techniques are geometric transformations, color space transformations, kernel filters, random erasing, and mixing images. However, data augmentation alone does not solve the fundamental problem low number of defective samples. Also, data augmentation is not helpful when the same defect occurs in a different region of the part or different variation of the same defect occurs. Moreover, the model is still bound to the samples that it has seen and trained on. Commonly, data augmentation is applied when there is a low amount of data (e.g., less than 1000 images) but even this amount of data is often difficult to acquire in a real manufacturing setting. Improved systems are desirable.SUMMARY

[0006] Briefly, aspects of the present disclosure address at least some of the above-mentioned technical problems pertaining to availability of defect data for training a machine learning model for vision-based inspection in industrial use-cases. Disclosed aspects provide a simulation-based synthetic defect generation framework to create a dataset for training a machine learning model for vision-based quality inspection of parts manufactured on a production line.

[0007] A first aspect of the disclosure provides a computer-implemented method for supporting vision-based inspection of manufactured parts on a production line. The method comprises creating a defect library in simulation utilizing historically acquired real images of manufactured part samples with defects, by digitization and parameterization of enclosed defect geometries of a number of defect types obtained from the real images. The method further comprises generating synthetic defects by applying randomization to parameters defining the enclosed defect geometries of the defect types in the defect library. The method further comprises carrying out a 3D superimposition of the synthetic defects at randomized positions on a CAD model of a target part to generate a number of defect model objects, wherein each defect model object includes a digital 3D representation of the target part containing one or more of the synthetic defects. The method further comprises rendering simulated camera images using the generated defect model objects and a digital twin of an inspection environment having one or more cameras, to create a dataset for training a machine learning model for vision-based inspection of manufactured parts on the production line.

[0008] A further aspect of the disclosure provides a method for carrying out online vision-based inspection of manufactured parts on a production line. The method comprises acquiring an image of a manufactured part via a camera in an inspection environment. The method further comprises processing the acquired image using a machine learning model to infer a defect in the manufactured part, wherein the machine learning model is trained in a supervised learning process on a dataset created using the above-described methodology.

[0009] Other aspects of the disclosure implement features of the above-described methodology in computer program products and computing systems for vision-based inspection of manufactured parts on a production line.

[0010] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0012] FIG. 1 is a schematic diagram illustrating a pipeline for generating a synthetic dataset for training a machine learning model to perform vision-based quality inspection.

[0013] FIG. 2 illustrates an example of a quality alert list obtained from a production line showing real images of parts with defects.

[0014] FIG. 3 illustrates an example of a parameterization of a defect geometry as a polyline, from a real image of a defective part.

[0015] FIG. 4 is a flowchart illustrating a process for defect randomization and superimposition based on extrusion of defect geometry from a solid model, according to a first embodiment.

[0016] FIG. 5 is a flowchart illustrating a process for defect randomization and superimposition based on application of a polytrim function on a mesh structure derived from a solid model, according to a second embodiment.

[0017] FIG. 6 illustrates randomly generated defects on a solid model of a sheet metal body.

[0018] FIG. 7 shows an example of a simulated camera image of a sheet metal body generated by an image Tenderer using defect model objects.

[0019] FIG. 8 is a flowchart illustrating a process for labeling defects in a simulated camera image according to an example embodiment.

[0020] FIG. 9 is a schematic diagram illustrating a vision-based inspection system for a production line using a machine learning model trained using the synthetic dataset, according to an embodiment.

[0021] FIG. 10 shows a computing system that can support vision-based inspection of manufactured parts on a production line according to disclosed embodiments.DETAILED DESCRIPTION

[0022] Defect data availability in manufacturing is a leading challenge in visual quality inspection on production lines. The disclosed methodology solves the limited defect data availability problem in defect detection tasks by using simulation as a main data source. The disclosed methodology leverages simulation technologies to generate, parameterize and randomize defects and render synthetic images to create dataset for training a machine learning (ML) model. Embodiments of the disclosed methodology are geared toward detection of surface-level and texture-level defects. Examples of such defects include scratches, cracks, splits, dents (low spots), bumps (high spots), and so on.

[0023] In an example manufacturing use-case, parts may be manufactured by different manufacturing processes upstream on a production line, such as stamping, die-casting, injection molding, etc., and transferred to an inspection environment by a gantry, a robotized system, or anyother type of transfer mechanism. The inspection environment may include an inspection station, for example, including a conveyor, on which a part to be inspected is placed. One or more cameras may be arranged in the inspection environment, which may be triggered to capture respective images of the part when the part is moved into a pre-defined position by the conveyor. In other arrangements, the inspection station may be stationary where there is no motion of the part during inspection and the inspection environment may comprise one or more robots capable of manipulating the position of the one or more cameras.

[0024] The inventive methodology disclosed herein includes creation of a defect library in simulation, which relies on the actual physical defects that occur on the production line. This ensures that the generated defects are realistic geometrically as well as visually. The actual physical defects may be derived from historically acquired real images of defective parts, which may be obtained from existing quality related documentation concerning the production line. Using the historically acquired real images, defects may be localized and parameterized (e.g., using polylines) to create a digitized defect library. The disclosed methodology then carries out a randomization of the parameters in the defect library (e.g., by polyline scaling) to generate randomized defect geometries, followed by a 3D superimposition of the generated defect geometry at randomized positions on a CAD model of the target part (e.g., by extrusion or mesh trimming), to create a number of CAD representations that define defect model objects. The methodology uses image rendering techniques to render a number of simulated camera images using the generated defect model objects. The image rendering is based on realistic camera and environment simulation using a digital twin of the inspection environment, and in some embodiments, may also involve additional randomization of environmental and operational conditions on the shop floor for better data variation. The rendered images are used to create a dataset for training a ML model for vision-based inspection of manufactured parts on the production line.

[0025] Turning now to the disclosed embodiments, FIG. 1 illustrates an example pipeline 100 for generating a synthetic dataset for training a ML model to perform vision-based quality inspection in a production line. The various modules, such as the defect simulator 104, the defect randomizer and superimpose 110 and the image Tenderer 114, described herein, including components thereof, may be implemented by a computing system in various ways, for example, as hardware and programming. The programming for the modules 104, 110 and 114 may take the form of processor-executable instructions stored on non-transitory machine-readable storage mediums and the hardware for themodules may include processors to execute those instructions. In embodiments, the computing system may suitably comprise a workstation having multiple graphics processing units (GPU).

[0026] The defect simulator 104 is configured to create a defect library 108 in simulation utilizing historically acquired real images 102 of manufactured part samples with defects. Usually, in a production line, every unique defect is recorded and tracked. For example, after a new defect occurs, a quality advisor may store the defective part sample, take images from multiple perspectives, and document the defect as well as the process characteristics. This document is usually referred to as a quality alert list, which can be used, for example, to train new operators or identify a root-cause of the problem if a similar defect happens again. An exemplary view of a quality alert list 200 is shown in FIG. 2. As shown, for a given part sample, a number of real images 102 are acquired from different perspectives where the defects are subsequently manually labeled, as shown by circles 202. Desirably, the images 102 may be acquired via one or more cameras in the inspection environment of the production line. In some embodiments, the images 102 may be acquired from multiple perspectives by an operator using a hand-held camera. Quality alert lists can be a powerful tool because they provide a good understanding about all possible defects that happened during the production process. Such a report can be utilized to infer high-risk regions on a part, defect geometry (size, length, depth, etc.) and defect types, as described below.

[0027] Continuing with reference to FIG. 1 , the defect library 108 is created by the defect simulator 104 by digitization and parameterization of enclosed defect geometries of a number of defect types obtained from the historically acquired real images 102. In one embodiment, the defect simulator 104 may be implemented using a computer aided design (CAD) software, such as NX™ developed by Siemens Industry Software, Inc., among others. To create the digital defect library 108, each defect on a real image 102 may be digitally labeled to parameterize a geometry enclosed by the defect. According to disclosed embodiments, the enclosed defect geometry may be parameterized as a polygon or polyline. Since an enclosed geometry is considered, the terms “polygon” and “polyline” may be used interchangeably. Polylines can be used to effectively parameterize any closed organic shape. In some scenarios, an enclosed defect geometry may be parameterized using regular shapes, such as circle, ellipse, etc.

[0028] A polyline may be generated by creating a segmentation around a defect, for example using available open-source labeling tools (e.g., in python). In some embodiments, a defect may be labeledby drawing an outline around the digitized defect using a sketch generator and transforming the sketch to a polyline. To illustrate an example of polyline parameterization, reference may be made to FIG. 3, where, on left is shown a real image 102 of a part 302 with a manually labeled defect 304 which can be obtained from a quality alert list. On the right is shown the same image 102, where the enclosed geometry of the defect is digitally labeled and parameterized as a polyline 306.

[0029] According to a disclosed embodiment as shown in FIG. 1, the real images 102 may be registered with a CAD model 106 of the respective manufactured part to generate a global mapping of a 2D pixel space of the real images 102 to a 3D space of the CAD model 106. The global mapping may be used to transform the parameterized enclosed defect geometries (e.g., polylines) from the 2D space of the real images 102 to the 3D space in the CAD model 106. Thereby, the actual dimensions and position of each known physical defect may be determined in simulation and stored in the defect library 108 (e.g., as coordinates of polygon vertices). Note that, depending on the part surface containing the defect (e.g., curved or plane surface), the transformed polyline on the CAD model 106 can be a 2D polyline or a 3D polyline. Each 2D or 3D polyline shape may be stored as a respective defect type in the defect library 108. The global positions of all the known defects on the CAD model 106 can be used to infer high-risk regions on the respective part. In embodiments, the defect library 108 may be created in simulation using real images 102 and CAD models 106 of a number of different types of parts produced on a single or multiple production lines on the shop floor.

[0030] The main motivation for creating a digital defect library 108 is to parameterize all known defect geometries. Doing so provides the capability to manipulate and alter the defect geometries and generate different types of defects. From smaller to larger defects, it may be possible cover a wide range of possibilities and combinations, which can make the ML model much more robust against different unknown defects. Parameterization is also an enabler to randomize the defect geometries, which can help to create a large, labeled dataset.

[0031] The above-described functionality can be realized via the randomized defect generator 110, in broadly two stages. A first stage involves generating synthetic defects by application of randomization to parameters defining the enclosed defect geometries of the defect types in the defect library 108. Within the first stage, two randomizations may be applied to each parameterized defect, namely, (1) geometry randomization and (2) position randomization. A second stage involves carrying out a 3D superimposition of these synthetic defects at the randomized positions on a CAD model 106of the target part to generate a number of defect model objects 112. Each defect model object 112 includes a digital 3D CAD representation of the target part containing one or more of the synthetic defects. In embodiments, the target part may have the same nominal geometry as the defective part samples (or at least some of them) from which the real images 102 were acquired.

[0032] According to a disclosed embodiment, the geometry randomization may involve, for each parameterized defect type in the defect library 108, scaling the outer edges of the enclosed defect geometry (polyline) along a length direction and / or width direction of the defect, to generate synthetic defects with randomized enclosed geometries. Scaling the enclosed defect geometry along the length of the defect can result in different sizes of a given defect type. Doing so along the width of the defect can help generate defects with thicker or thinner geometry. Additionally, or alternately to the scaling, geometry randomization may also involve, for each parameterized defect type in the defect library 108, rotating the enclosed defect geometry (polyline), to generate synthetic defects with randomized angular orientations. It has been realized that rotational randomization can make the ML model much more robust against the different part orientations. Of course, these perturbations may change the geometry. However, this geometry change can help the ML model to have more reliable and generic capabilities. An example of synthetic defect generation based on geometry randomization (by scaling and rotation) can be visualized in EIG 6.

[0033] A second randomization may be applied on the position of the defects. As described above, the defect library 108 may be used to infer high-risk regions on a target part where defects are most likely to occur. According to a disclosed embodiment, position randomization may be implemented by generating randomized positions for the synthetic defects based on the identified high-risk regions on the target part, such that in any defect model object 112, a predefined majority of the synthetic defects (e.g., 60-80%) lie within the identified high-risk regions. Doing so ensures that in addition to the high-risk regions, defects are also randomly positioned in other locations to increase the dataset size. It is important to note that position randomization may be applied based on the corners, edges and flat surfaces in their domain-specific context. Lor example, defects resulting from a stamping process may usually occur in the part edges and curvatures, while paint defects may usually occur along the cracks and crevasses.

[0034] The second stage of operation of the randomized defect generator 110, namely, carrying out a 3D superimposition of the generated synthetic defects on the target part, can be implemented ina number of ways. A first disclosed embodiment involves a 3D superimposition by extrusion of the enclosed effect geometry (polyline) of a generated synthetic defect from a surface of a solid body representation of the CAD model 106 of the target part. Extrusion refers to a process of moving a uniform a cross-section (e.g., a polygon) along a path in a 3D space. In some embodiments, the path may be normal to the datum plane where the defect is to be superimposed. In the present context, the extrusion of the enclosed defect geometry may be carried out additively (e.g., in case of a specular bump or dirt defect on paint) or subtractively (e.g., in case of a dent or scratch), at positions that are determined by the position randomization at the previous stage. The height of the extruded geometry of the synthetic defect may be randomly defined so as to he within a specified height range, e.g., to ensure that a dent does not exceed a local thickness of the part, or that a bump is not unrealistically tall.

[0035] The above-described technique of extrusion can be used to create very realistic cracks, splits, scratches and any type of hole defects, which can lead to a robust ML model. In the case of superimposing scratch defects, after the cropping is applied, it usually results in clean-cut depth edges on the surface. Also, the scratch surface typically shows up as a clean, smooth surface. In one embodiment, a randomized texture may be added to an extruded defect surface in case of a subtractive extrusion (especially for scratch defects), e.g., using a CAD editor. This operation can ensure that the defect is realistic and detectable by the ML model.

[0036] EIG. 4 illustrates an example of a process 400 for defect randomization and superimposition based on extrusion of defect geometry from a solid CAD model of a target part. The process 400 can be embodied as a computer-implemented routine. The process 400 includes selecting a datum plane (block 402), which may be defined by a surface on the solid model where a defect is to be superimposed. The process 400 accesses the defect library (block 404) to select a defect type parameterized by a polyline. Lor the selected defect type, the process 400 applies polyline scaling (block 406) for geometry randomization and generates each randomized defect geometry as a sketch (block 408). The process 400 also accesses high-risk regions (block 410), which may be inferred from the defect library or otherwise specified by a user. Based on the high-risk regions, the process 400 generates randomized defect positions (block 412), ensuring that a predefined majority of the defects lie within the high-risk regions. The process 400 then loops between the randomized positions, such that at each randomized position, a randomized defect geometry is extruded from the solid model (block 414) and the resulting solid model is saved as a defect model object, e.g., in OBJ format (block416). After executing a defined number of inner loops between the positions, the process 400 returns to block 404 to execute the above-described steps for a different type of defect selected from the defect library. The process 400 continues in this manner for a defined number of outer loops between defect types. At the completion of the loops, a number of defect model objects (OBJ files) are generated, each including a 3D solid representation of the target part containing randomized synthetic defects.

[0037] According to a second disclosed embodiment of 3D superimposition, the solid body representation of the CAD model of the target part is first tessellated to generate a mesh structure. In a mesh structure, the part surface is described in 3D space by a collection of polygonal surface elements that are connected by their common edges and / or vertices. The parameterized enclosed defect geometry (polyline) of each generated synthetic defect is then applied as a mesh trimming function with a defined height, at a position determined by the position randomization in the previous stage. The mesh trimming function removes surface elements in the mesh structure that are located in the 3D space enclosed by the polyline (characterized by the defined height). This approach is relatively straightforward to apply and can be efficient especially for creating split and crack defects.

[0038] FIG. 5 illustrates an example of a process 500 for defect randomization and superimposition based on application of a polytrim function on a mesh structure derived from a solid CAD model of a target part. The process 500 can be embodied as a computer-implemented routine. The process 500 starts with transformation of the solid CAD model to a mesh structure by tessellation (block 502). The process 500 accesses the defect library (block 504) to select a defect type parameterized by a polyline. For the selected defect type, the process 500 applies polyline scaling (block 506) for geometry randomization and generates polyline trim objects (block 508) for the randomized geometries. The process 500 also accesses high-risk regions (block 510), which may be inferred from the defect library or otherwise specified by a user. Based on the high-risk regions, the process 500 generates randomized defect positions (block 512), ensuring that a predefined majority of the defects he within the high-risk regions. The process 500 then loops between the randomized positions, such that at each randomized position, a generated polyline trim object is applied to the mesh (block 514) and the resulting mesh is saved as a defect model object, e.g., in OBJ format (block 516). After executing a defined number of inner loops between the positions, the process 500 returns to block 504 to execute the above-described steps for a different type of defect selected from the defect library. The process 500 continues in this manner for a defined number of outer loops between defect types. At the completion of the loops, a number of defect model objects (OBJ files) are generated, each including a 3D mesh representationof the target part containing randomized synthetic defects.

[0039] FIG. 6 illustrates randomly generated defects superimposed on a solid model of a sheet metal body. The image (A) shows a sheet metal body 600 having split and crack defects 602 at the edges, hole defects 604, bumps 606 and scratch defects 608. The position and geometry of these defects may be extracted from the defect library. The image (B) illustrates randomly generated geometries of the split and crack defects 602’, hole defects 604’ and bumps 606’ superimposed on the sheet metal body 600 by extrusion. The image (C) illustrates randomly generated geometries of the scratch defects 608’ superimposed on the sheet metal body 600 by extrusion. As seen, the disclosed methodology can be used to control the size, geometry, position and type of defects superimposed on a part. Although these defect geometries are not coming from a physics-based model (e.g., based on finite element analysis), they can provide enough information to state-of-the-art deep learning models to generalize over real parts.

[0040] Turning back to FIG. 1, the defect model objects 112 (e.g., OBJ files) generated using the randomized defect generator 110 are fed into an image Tenderer 114. The image Tenderer 114 may comprise a 3D simulation engine that can render a number of simulated camera images 124 utilizing, in each case, one of the defect model objects 112. In particular, as shown, the image Tenderer 114 may include a camera simulator 116 that can simulate the one or more cameras of the inspection environment based on their position (e.g., location, angle, etc.) and camera parameters (e.g., exposure, color mode, resolution, analog gain, field of view or FOV, depth of focus, etc.) and an environment simulator 118 that can simulate the inspection environment (e.g., including the conveyor, ambient lighting, etc.).

[0041] A realistic simulation can be performed by the camera simulator 116 and environment simulator 118 using a digital twin 120 of the actual inspection environment. The digital twin 120 may comprise a 3D model of the actual inspection system, for example, including the conveyor and other components of the transfer mechanism. The digital twin 120 may be created based on an overall system design, for example, using information pertaining to the cameras, line configuration and process parameters, among others. The camera information may include information such as location and design of the fixture to mount the cameras (e.g., array or grid, distance from inspection station, etc.), available number of cameras, candidate camera trigger locations, camera parameters (such as mentioned above), among others. The line configuration information may include, for example,inspection station background, conveyor speeds, ambient lighting conditions, line / station dimensions, part orientation, etc. The process parameters may include parameters related to the production process, such as shift or time of day, number of operators, etc.

[0042] The image Tenderer 114 may be configured to generate, for a given defect model object 112, multiple simulated camera images 124 by simulating the inspection environment including the one or more cameras, based on randomization of a number of operational and environmental parameters. To that end, the image Tenderer 114 may include an environmental and operational randomizer 122 that can use a set-up defined by the digital twin 120 as a basis to generate randomized operational and environmental parameters. The camera parameters may be constrained using the digital twin 120 of the inspection environment.

[0043] Operational parameters randomized by the randomizer 122 may include, for example, at least some or all of: part orientation, part translation, conveyor speed, vibration in camera position (e.g., due to robot motion) and cycle time. Environmental parameters randomized by the randomizer 122 may include, for example, at least some or all of: conveyor dimensions, background (conveyor) color and texture, part color and texture and ambient lighting. The randomized parameters may be fed to the camera simulator 116 and the environment simulator 118 to generate a large number of simulated camera images 124 for each defect model object 112. Each rendered image 124 may be identified by its respective meta data, which may be defined by parameter values of the randomized environmental and operational parameters.

[0044] The disclosed environmental and operational randomization is distinct from generic domain randomization approaches. Traditionally, the approach used in domain randomization has been to place the cameras in different (and in most cases unrealistic) positions to increase the data availability and variation in the dataset. Even though, this approach works quite well for generic object detection tasks (e.g., pick and place applications), it often does not translate to visual quality inspection usecases. The main reason is that the objects (defects in this context) in visual quality inspection tasks are much smaller than the overall image size (for a 3mm defect, only 10 to 100 pixels contain the defect information in a camera with 16 MP resolution). Therefore, if no constraint is applied on the camera positions and its FOV, there is a high possibility that most of the defects will be out-of-focus or not even present in the simulated images. Accordingly, in the disclosed embodiment, as a first step, the inspection environment set-up is created in simulation, and subsequently that set-up is used as abasis for environmental and operational randomization.

[0045] Desirably, the image Tenderer 114 may be configured to generate the simulated camera images 124 as photorealistic images. A photorealistic image is an image rendering that is based on simulation of the behavior of light, for example, using techniques such as ray tracing. A photorealistic image may thus have realistic lighting reflections as can be expected on a real shop floor. The photorealistic images may typically comprise RGB images (having red, green and blue channels of pixel intensities). In some embodiments, a real background texture may be overlaid on the photorealistic camera images 124.

[0046] In a non-limiting example implementation, the inspection environment including the camera set-up can be generated and validated in simulation using a manufacturing process simulation engine, such as Process Simulate™ developed by Siemens Industry Software, Inc. The generated virtual set-up may then be exported to a photorealistic rendering engine with ray tracing capabilities, such as Omniverse™ developed by Nvidia Corporation, to provide an easy to use and accurate representation of lighting and camera conditions in the inspection environment. A custom tool built on the Omniverse™ Kit and Replicator can be used to randomize environmental and operational parameters for each defective part and render photorealistic images as captured by cameras in the inspection environment. In this case, it is also possible to use the real background from the inspection environment and overlay on the synthetic images.

[0047] An example of a photorealistic camera image 124 generated by the image Tenderer 114 according to the disclosed implementation (converted to grayscale herein) is shown in FIG. 7. The image 124 shows a simple example of a blank sheet metal part 700. As shown, the rendered representation of the defective part 700 includes a high spot or bump 702 which may be caused, for example, due to a dent in the back of the sheet metal. As shown in the zoomed-in view of the defective region 706, the photorealistic image 124 also depicts a shadow 704 caused by the bump 702. The rendered representation is based on randomization of defect parameters and positions of actual defects digitized and stored in the defect library 108 and superimposition of the randomized defect geometries on a CAD model of the part, as described above.

[0048] To create a labeled dataset for supervised learning, the simulated camera images 124 rendered by the image Tenderer 114 may be attached with labels or annotations (e.g., bounding boxes)localizing the defects on the rendered images 124. The labeling process can be carried out manually, semi-automatically or in a fully automated manner.

[0049] FIG. 8 illustrates an example of an automated process 800 for labeling defects in a simulated camera image. The process 800 leverages information about the global position of the generated defects based on the CAD model (block 802). For example, the global position may be defined by 3D coordinates of the vertices of the polygon representing the defect in the CAD model. In particular, this information may be extracted from the defect model objects utilized by the image Tenderer to generate the simulated camera images. The process 800 then includes performing a coordinate transformation (block 804) to map the 3D coordinates of the defects (polygon vertices) from the CAD model into a 2D pixel space. Having determined the coordinate transformation, the process 800 takes as input the simulation camera FOV and the resolution from the image Tenderer (block 806), applies the coordinate transformation (block 804) and generates a defect annotation for the rendered image (block 808), for example, in terms of coordinates (max and min) of a 2D bounding box, for each defect in the camera FOV. Note that, using this approach, the same label or annotation may be attached to each of the randomized images that is rendered based on a given defect model object and camera FOV.

[0050] Referring to FIG. 1, the simulated camera images 124 generated by the image Tenderer 114, along with the attached labels or annotations (e.g., bounding boxes), may be used to create a synthetic dataset for training a ML model for vision-based inspection of manufactured parts on a production line. In embodiments, the ML model may include a deep neural network (DNN), especially a deep convolutional network, among others. The ML model may be trained in a supervised learning process, by inputting the images 124 (which may define a DNN input layer) and comparing a predicted output of the model (which may define a DNN output layer) against the annotations as ground truth to compute a loss function. The learning process may comprise iteratively adjusting learnable model parameters (e.g., DNN weights and biases), for example, using a method of gradient descent, over a set of images 124, such that the loss function is minimized. The set of images 124 used for model training may pertain to different parts in multiple inspection environments. In some embodiments, the ML model may be pretrained on an available image database for object recognition, such as ImageNet, and fine-tuned to the specific industrial use-case using the dataset generated by the disclosed methodology. After training, the ML model may be validated using real images of parts on the production line.

[0051] Industrial cameras used for vision-based quality inspection usually have high resolutions, typically ranging from 16 megapixels to 24 megapixels, which can lead to a large amount of data. However, depending on their architecture, industrial deep learning models typically work on low or medium resolution images. Resizing of a high-resolution image to a low-resolution image can lead to loss of critical information, since many of the defects can be very small (e.g., 1 mm or less). Moreover, there is high correlation between the model input size and its complexity. Considering that industrial inspection use-cases are required to achieve a low cycle time, it is desirable for the model to have low complexity. To address the above challenges, in some embodiments, a data pre-processing step may be introduced to extract smaller patches (having reduced number of pixels) from high-resolution images 124, so that the extracted patches can be ingested and processed efficiently by the ML model. The patches may be extracted with a defined patch size in an ordered or randomized manner. The generated labels or annotations may be adjusted and assigned to the respective patches in the images where they are located. The patches with the assigned annotations may now form the synthetic dataset for training the ML model.

[0052] FIG. 9 shows an example of a vision-based quality inspection system 900 for a production line according to an example embodiment. The system 900 includes an online defect detector module (inference system) 910 deploying a ML model 912, which was trained using a synthetic dataset as described above. The defect detector module 910, including components thereof, may be implemented by a computing system, such as an Edge device. The programming for the module may take the form of processor-executable instructions stored on non-transitory machine-readable storage mediums and the hardware for the module may include processors to execute those instructions.

[0053] Referring to FIG. 9, the system 900 may comprise one or more cameras 906 arranged in an inspection environment 902 of a production line. The production line may comprise a queue of manufactured parts 904 that may be transported to the inspection environment 902 by a conveyor. In examples, the manufactured parts 904 may comprise die-cast parts, stamped metal parts, injection molded plastic parts, among others. During operation, each of the one or more cameras 906 may be triggered to acquire one or more real images 908 of each manufactured part 904. The defect detector 910 may process an input based on the acquired real images 908 and generate an inference regarding the presence and location of a defect using the trained ML model 912.

[0054] Consistent with some embodiments, the system 900 may include a patch extractor module(not shown) configured to extract patches from each acquired real image 908. The extracted patches for each image 908 may comprise, in this case, ordered patches with a defined patch size. The extracted patches may be provided to the defect detector 910, which may use the ML model 912 to detect and localize defects on the patches. Ordered patches of an image may then be stitched together to recreate the whole image with annotations or bounding boxes around the detected defects, to produce an output image 914. In certain embodiments, depending on the resolution of the images 908 and / or the architecture of the ML model 912, the input to the defect detector 910 may comprise whole images 908. A patch extractor may be obviated in this case.

[0055] Based on the output of the defect detector 910, appropriate control action may be executed. Examples of control action may include, for example, isolating a defective part from production, shutting down production, and so on. The control action may be executed automatically, semi- automatically, or manually, responsive to the output of the defect detector 910, specifically, a positive defect detection in one or more manufactured parts 904 in the production line. For example, in one embodiment, as shown in FIG. 9, the system 900 may include a field controller 916 for automatically executing a control action to control the production line based on the output of the defect detector 914. In embodiments, the field controller 916 may comprise or be integrated with automation system equipment such as PLCs (Programmable Logic Controllers), drives, etc. for executing the control action. Additionally, or alternately, the output of the defect detector 910 (e.g., presence and location of identified defects) may be visualized and displayed to an operator via an HMI device 918, to enable an operator to initiate a control action.

[0056] FIG. 10 shows an example of a computing system 1000 that can support vision-based inspection of parts on a production line according to disclosed embodiments. In examples, the computing system 1000 may be configured as a powerful multi-GPU workstation, among other types of computing devices. The computing system 1000 includes at least one processor 1010, which may take the form of a single or multiple processors. The processor(s) 1010 may include a one or more CPUs, GPUs, microprocessors, or any hardware devices suitable for executing instructions stored on a memory comprising a machine-readable medium. The computing system 1000 further includes a machine-readable medium 1020. The machine-readable medium 1020 may take the form of one or more media including any non-transitory electronic, magnetic, optical, or other physical storage device that stores executable instructions, such as defect simulation instructions 1022, randomized defect generation instructions 1024 and image rendering instructions 1026, as shown in FIG. 10. Assuch, the machine-readable medium 1020 may be, for example, Random Access Memory (RAM) such as a dynamic RAM (DRAM), flash memory, spin-transfer torque memory, an Electrically- Erasable Programmable Read-Only Memory (EEPROM), a storage drive, an optical disk, and the like.

[0057] The computing system 1000 may execute instructions stored on the machine-readable medium 1020 through the processor(s) 1010. Executing the instructions (e.g., the defect simulation instructions 1022, the randomized defect generation instructions 1024 and the image rendering instructions 1026) may cause the computing system 1000 to perform any of the technical features described herein, including according to any of the features of the defect simulator 104, the randomized defect generator 110 and the image Tenderer 114 described above.

[0058] The systems, methods, devices, and logic described above, including the defect simulator 104, the randomized defect generator 110 and the image Tenderer 114, may be implemented in many different ways in many different combinations of hardware, logic, circuitry, and executable instructions stored on a machine-readable medium. For example, these modules may include circuitry in a controller, a microprocessor, or an application specific integrated circuit (ASIC), or may be implemented with discrete logic or components, or a combination of other types of analog or digital circuitry, combined on a single integrated circuit or distributed among multiple integrated circuits. A product, such as a computer program product, may include a storage medium and machine-readable instructions stored on the medium, which when executed in an endpoint, computer system, or other device, cause the device to perform operations according to any of the description above, including according to any features of the defect simulator 104, the randomized defect generator 110 and the image Tenderer 114. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network.

[0059] The processing capability of the systems, devices, and engines described herein, including the defect simulator 104, the randomized defect generator 110 and the image Tenderer 114, may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, andmay be implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs may be parts (e.g., subroutines) of a single program, separate programs, distributed across several memories and processors, or implemented in many different ways, such as in a library (e.g., a shared library).

[0060] Although this disclosure has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the patent claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for supporting vision-based inspection of manufactured parts on a production line, comprising: creating a defect library in simulation utilizing historically acquired real images of manufactured part samples with defects, by digitization and parameterization of enclosed defect geometries of a number of defect types obtained from the real images, generating synthetic defects by applying randomization to parameters defining the enclosed defect geometries of the defect types in the defect library, carrying out a 3D superimposition of the synthetic defects at randomized positions on a CAD model of a target part to generate a number of defect model objects, wherein each defect model object includes a digital 3D representation of the target part containing one or more of the synthetic defects, and rendering simulated camera images using the generated defect model objects and a digital twin of an inspection environment having one or more cameras, to create a dataset for training a machine learning model for vision-based inspection of manufactured parts on the production line.

2. The method according to claim 1, wherein the defect library is created by: digitally labeling defects on the historically acquired real images of the manufactured part samples to parameterize a geometry enclosed by the labeled defects, and transforming the parameterized enclosed defect geometries from a 2D space of the real images to a 3D space in a CAD representation of the manufactured part samples.

3. The method according to any of claims 1 and 2, wherein the enclosed defect geometries are parameterized as polylines.

4. The method according to any to any of claims 1 to 3, wherein the synthetic defects are generated by randomizing, for each defect type in the defect library, the parameters of the enclosed defect geometry, by scaling outer edges of the enclosed defect geometry along a length direction and / or width direction and / or by rotating the enclosed defect geometry.

5. The method according to any of claims 1 to 4, comprising generating randomized positions for the synthetic defects based on an identified high-risk region, such that a predefined majority of the synthetic defects lie within the identified high-risk region.

6. The method according to any of claims 1 to 5, wherein the target part has the same nominal geometry as at least some of the manufactured part samples with defects, the real images of which are utilized to create the defect library.

7. The method according to any of claims 1 to 6, wherein the 3D superimposition comprises performing an additive or subtractive extrusion of the parameterized enclosed defect geometry of a generated synthetic defect from a surface of a solid body representation of the CAD model of the target part, at a position determined by the position randomization.

8. The method according to claim 7, wherein the 3D superimposition comprises randomly defining a height of the extruded geometry of the synthetic defect within a specified height range.

9. The method according to any of claims 7 and 8, further comprising, applying a randomized texture to an extruded defect surface in case of a subtractive extrusion.

10. The method according to any of claims 1 to 6, wherein the 3D superimposition comprises: tessellating a solid body representation of the CAD model of the target part to generate a mesh structure, applying the parameterized enclosed defect geometry of a generated synthetic defect as a mesh trimming function with a defined height, at a position determined by the position randomization.

11. The method according to any of claims 1 to 10, wherein the simulated camera images are rendered based on a randomization of a number of environmental and operational parameters, and wherein camera parameters are constrained using the digital twin of the inspection environment.

12. The method according to claim 11, wherein the randomized environmental and operational parameters are selected from the group consisting of: part orientation, part translation, conveyor speed, vibration in camera position, cycle time, conveyor dimensions, background color and texture, part color and texture, and ambient lighting.

13. A method for carrying out online vision-based inspection of manufactured parts on a production line, comprising: acquiring an image of a manufactured part via a camera in an inspection environment, and processing the acquired image using a machine learning model to infer a defect in the manufactured part, wherein the machine learning model is trained in a supervised learning process on a dataset created by performing a method according to any of claims 1 to 12.

14. A non-transitory computer-readable storage medium encoded with instructions that, when processed by a computing system, configure the computing system to perform the method according to any of claims 1 to 13.

15. A computing system comprising: one or more processors, and non-transitory memory storing algorithmic modules executable by the one or more processors to carry out a method according to any of claims 1 to 13.

Images