Image acquisition system optimization in vision-based industrial automation

Abstract

According to disclosed embodiments for configuring an image acquisition system with one or more cameras for vision-based inspection of parts on a production line, the simulation engine renders a composite image of the part on the production line acquired by the one or more cameras based on the 3D model of the part and a configuration of the image acquisition system defined by the optimizable parameters. The surface coverage measurement engine uses the output of the simulation engine to measure blind spots on the surface of the part for individual cameras and thereby determine a measurement of visible surface coverage on the 3D model of the part. The optimization engine generates an updated configuration of the image acquisition system by updating optimizable parameters based on an evaluation of an optimization objective defined by the measured visible surface coverage. The above process is iteratively performed to determine a final configuration of the image acquisition system.

Description

Technical Field

[0001] The present disclosure relates to computer vision systems for industrial automation applications. Embodiments of the present disclosure relate specifically to techniques for optimizing the configuration of an image acquisition system for vision-based inspection of parts on a production line. Background Art

[0002] Computer vision has become central to various parts of the modern manufacturing process. For example, computer vision techniques can assist in visual quality tasks, which are becoming increasingly important in reducing the number of defective products. One challenge in successfully deploying these advanced technologies is correctly determining the configuration of the image acquisition system during the inspection process, such as the number of cameras, camera positions and angles, etc. The performance of artificial intelligence (AI) models can depend heavily on the quality of the image data they are trained on. Therefore, if the initial system design is not done correctly (for example, the camera is located in the wrong location or the wrong lighting scheme is used), the quality of the image data can be degraded adversely.

[0003] Currently, the installation or even modification of image acquisition systems for production line inspection can be a tedious manual process that often requires full-time human expertise from different fields, such as lighting, camera, AI, and production experts. In addition, the design period of most vision projects may involve a comprehensive experimentation phase, where domain experts may try different configurations (e.g., number of cameras, camera positions and angles, etc.) to design the image acquisition system. However, the final design based on the configuration parameters determined by the experts is often subjective and produces suboptimal results. For example, the design may only be suitable for a specific part, rather than being general enough. Summary of the invention

[0004] In short, aspects of the present disclosure provide a virtual commissioning framework to optimize the configuration of an image acquisition system used for vision-based inspection of parts on a production line.

[0005] A first aspect of the present disclosure provides a computer-implemented method for configuring an image acquisition system, the image acquisition system comprising one or more cameras for vision-based inspection of parts on a production line. The method includes running a simulation engine for rendering a composite image of a part on a production line acquired by one or more cameras based on a 3D model of the part and a configuration of the image acquisition system defined by optimizable parameters. The method also includes running a surface coverage measurement engine for measuring blind spots on the surface of the part for a separate camera in one or more cameras using the output of the simulation engine, and thereby determining a measurement of visible surface coverage on the 3D model of the part. The method also includes executing an optimization engine for updating the optimizable parameters based on an evaluation of an optimization goal defined by the measured visible surface coverage to generate an updated configuration of the image acquisition system. The method includes performing the above activities in multiple iterations to determine the final configuration of the image acquisition system for vision-based inspection on a production line.

[0006] Other aspects of the present disclosure implement features of the above methods in computing systems and computer program products for configuring an image acquisition system for vision-based inspection of parts on a production line.

[0007] Additional technical features and benefits can be achieved through the technology of the present disclosure. The embodiments and aspects of the present disclosure are described in detail herein and are considered to be part of the claimed subject matter. For a better understanding, reference is made to the detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing and other aspects of the present disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings. To easily identify the discussion of any element or action, the most significant digit or digits in the reference numerals refer to the figure number in which the element or action is first introduced. For clarity, some of the images herein are schematically represented as line drawings.

[0009] Figure 1 An example of an inspection system for a production line is shown, the inspection system including an image acquisition system for vision-based inspection of parts.

[0010] Figure 2 is a schematic diagram of a system for configuring an image acquisition system for vision-based inspection of parts on a production line according to an example embodiment.

[0011] Figure 3 An example of a composite image of a part rendered by a simulation engine for a 2D image-based blind spot detector is shown.

[0012] Figure 4 The output of the global context mapper with a 2D image based blind spot detector is shown.

[0013] Figure 5 Schematic diagram of the measurement of the angle of a planar surface by a blind spot detector based on 3D geometry.

[0014] Figure 6 The output of the global context mapper with a 3D geometry based blind spot detector is shown.

[0015] Figure 7 is a schematic diagram of deployment of the disclosed method on a vision-based inspection system.

[0016] Figure 8 A computing system is shown that can support configuration of an image acquisition system for vision-based inspection of parts on a production line according to example implementations. DETAILED DESCRIPTION

[0017] In the design of image acquisition systems for end-of-line inspection in production processes, the main goals are to acquire the best quality image data (such as having low light reflections, seeing the smallest defect size, robustness against motion blur, etc.), and to capture all parts on the inspection conveyor system. Full part coverage can be achieved, for example, by using several cameras and / or multiple trigger times based on the size of a specific part. This approach can bring maintenance challenges due to additional resource data, data and hardware management challenges due to increased hardware complexity, and bandwidth issues, and may not be universal for different types of parts.

[0018] One of the technical problems caused by uninformed design is the so-called blind spot. A blind spot is defined as an area on the surface of a part that is not visible to the camera lens due to the position of the camera relative to the part. The inventors have recognized that full part coverage using multiple cameras / trigger times as described above, for example, may not necessarily solve the problem of blind spots, as blind spots may even exist within the field of view of the camera.

[0019] The disclosed method can generate optimized configuration parameters for an image acquisition system via surface coverage measurements based on blind spot detection using simulation. The disclosed method can be implemented as part of a virtual commissioning pipeline to ensure optimal configuration for image acquisition in a physical environment without the need for real-world experiments.

[0020] Now turning to the attached figure, Figure 1An inspection system for a production line is shown, in which the disclosed method can be appropriately deployed. As shown, the inspection system 100 may include a conveyor 102 for moving a part 104 through an inspection area. The part 104 may be manufactured or assembled upstream of the production line and conveyed to the conveyor 102, for example, by a gantry, a robotic system, or any other type of conveying mechanism. When the part 104 is moved by the conveyor 102, an image acquisition system 106 may be triggered to acquire an image of the part 102 via one or more cameras. In the example shown, the image acquisition system 106 includes two cameras 108a, 108b mounted on a horizontal fixture 110 that extends across the width of the conveyor 102. The position of the cameras 108a, 108b can be adjusted by moving them along the horizontal fixture 110 (in this example by translating along the Y axis). The angles of the cameras 108a, 108b can be adjusted by rotation, such as by rolling, pitching, and yaw (in this example, by rotating around the X, Y, and Z axes, respectively). Additionally, depending on the size of the part 104, each of the cameras 108a, 108b may be triggered one or more times to capture the entire extent of the part 104. In various other embodiments of the inspection system, the inspection station may be stationary, wherein there is no movement of the part during inspection, and / or the image acquisition system may include one or more robots capable of manipulating the position and angle of one or more cameras.

[0021] Images captured by the image acquisition system 106 may be processed, for example, using an artificial intelligence (AI) model and / or a computer vision algorithm to detect the presence of defects in the part. To this end, the image acquisition system 106 may be optimally configured in terms of parameters such as the number of cameras, camera positions, angles, etc., which may be determined using the methods described herein.

[0022] Figure 2 An example implementation of a system 200 for configuring an image acquisition system for vision-based inspection of parts on a production line is shown. The various engines described herein (including the simulation engine 204, the surface coverage measurement engine 216, and the optimization engine 236) can be implemented by a computing system in various ways, such as hardware and programming. The programming for the engines 204, 216, and 236 can be in the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware for the engines can include a processor that executes these instructions. The processing capabilities of the systems, devices, and engines described herein (including the simulation engine 204, the surface coverage measurement engine 216, and the optimization engine 236) can be distributed among multiple system components, such as in multiple processors and memories that optionally include multiple distributed processing systems or cloud / network elements.

[0023] refer to Figure 2 , the simulation engine 204 can be operated to render a composite image of a part on a production line acquired by one or more cameras based on a 3D model of the part and a configuration of the image acquisition system defined by the optimizable parameters. The 3D model can include, for example, a CAD model of the part. The surface coverage measurement engine 216 can be operated to use the output 214 of the simulation engine 204 to measure blind spots on the surface of the part for a separate camera, and thereby determine a measurement 222 of visible surface coverage on the 3D model of the part. The optimization engine 236 can be operated to update the optimizable parameters based on an evaluation of the optimization goal defined by the measured visible surface coverage 222 to generate an updated configuration of the image acquisition system. The above steps can be performed starting from the initial configuration 212 of the image acquisition system through multiple iterations to iteratively determine the final configuration 248 of the image acquisition system for vision-based inspection on the production line. The initial configuration 212 can be specified, for example, based on input from a domain expert. When the optimization engine 236 meets the convergence criteria, the final configuration 248 can be determined.

[0024] In one implementation, the simulation engine 204 may use the camera simulator 206 to perform a simple camera simulation to render a composite image based on the 3D model of the part and the configuration parameters of the image acquisition system. The configuration parameters may include optimizable parameters, such as the number, position, and angle of cameras, etc., and fixed parameters, such as exposure, color mode, resolution, simulation gain, field of view, depth of focus, etc. In an embodiment, some of the so-called fixed parameters mentioned above may also be defined as optimizable parameters.

[0025] According to the disclosed embodiments, a more realistic simulation can be performed using a digital twin 202 of the inspection system. The digital twin 202 may include a 3D model of the actual inspection system, including, for example, a conveyor and other components of the conveying mechanism. The digital twin 202 may be created based on the overall system design, for example, using information about the image acquisition system, the parts to be inspected, the line configuration, and the process parameters. The image acquisition system information may include information such as the location and design of the fixtures for mounting the cameras (e.g., array or grid, distance from the inspection station, etc.), the number of available cameras, candidate camera trigger positions, camera fixing parameters (such as those mentioned above), etc. The part information may include a 3D model of each part to be inspected, such as a CAD model. The line configuration information may include, for example, the inspection station background, conveyor speed, ambient lighting conditions, line / station size, part orientation, etc. The process parameters may include parameters related to the production process, such as shifts or time of day, number of operators, etc. Having created the digital twin 202, the simulation may be started using an initial configuration 212 of the image acquisition system. The initial configuration 212 may include, for example, preferred initial values ​​of optimizable parameters specified by a domain expert.

[0026] Consistent with the disclosed embodiments, the simulation engine 204 may include an environment simulator 208 that may be used in conjunction with the camera simulator 206 to render multiple synthetic images for each individual camera based on randomization of environmental and operational parameters associated with the production line. The environment simulator 208 may utilize the digital twin 202 to generate a large number of scenarios by randomizing environmental and operational parameters such as ambient lighting conditions, part orientation, conveyor speed vibration, operators with adjacent inspection stations, conveyor / inspection station backgrounds, etc. The camera simulator 206 may be run to render synthetic images corresponding to each camera for each scenario generated by the environment simulator 208 using the optimizable parameters of the image acquisition system for a given iteration of the optimization. The randomization achieved by the environment simulator 208 may differ from traditional domain randomization in that it focuses on variables in the physical environment of the production line and may therefore simulate realistic variations that may be seen on an actual production line. The variations achieved by randomization may generalize camera positions and make the optimization more robust.

[0027] The surface coverage measurement engine 216 may include a blind spot detector 218, which may use the output 214 of the simulation engine 204 to measure blind spots on the surface of the part for each camera being simulated, and a global context mapper 220, which may map the blind spots of the individual cameras onto the 3D model of the part to determine a measure of the overall visible surface coverage. The measured visible surface coverage 222 may be used as a metric to optimize the camera configuration.

[0028] As stated above, blind spots associated with a camera can be defined as areas on the surface of a part that are not visible to the camera lens due to the position of the camera. For example, typically for a top-down camera, any vertical surface can be a potential blind spot. In image space, blind spots can appear as shadowed areas due to the reflection pattern from the part to the camera. In the present disclosure, two different modalities of blind spot detectors are described.

[0029] The first modality of the blind spot detector 218 may involve a 2D image based blind spot detector. In this case, the output 214 of the simulation engine 204 defining the input of the blind spot detector 218 may include a realistic image. The realistic image is an image rendering based on a simulation of the behavior of light using, for example, a technique such as ray tracing. Thus, the realistic image may have realistic lighting reflections, as may be expected on a real workshop. According to the disclosed embodiments, the realistic image may include an RGB image (red, green, and blue channels with pixel intensities).

[0030] Figure 3An example of a rendered image that may be processed by the 2D image-based blind spot detector 218 is schematically shown in . Here, image 300 corresponds to a portion of a part surface captured by a single camera with a top-down field of view (FOV). For clarity, image 300 is represented as a line drawing without photorealistic effects. However, in a realistic rendering of the image, surfaces perpendicular to the FOV plane, such as the interior vertical walls 302, 304, 306 in this example, would appear as shadowed areas. Such shadowed areas may have a different pixel color density than surfaces facing the camera FOV plane, such as surfaces 308, 310, 312. Therefore, there may be a one-to-one correlation between shadowed areas and blind spots.

[0031] To implement the 2D image based blind spot detector 218, a range of color density may be defined for classifying blind spots and visible surfaces. In some embodiments, multiple ranges of color density may be defined to even classify different levels of visible surfaces. Density may be defined as a measure of the "darkness" or "lightness" of an area in an image, which may be measured based on the RGB values ​​of a pixel. For each individual camera being simulated, the 2D image based blind spot detector 218 may segment the corresponding rendered realistic image based on a defined range of color density levels to generate a 2D surface overlay depicting visible areas from shadow areas corresponding to blind spots.

[0032] In an embodiment, the 2D image-based blind spot detector 218 may operate as follows. At the beginning, each image may be processed to remove the background from the image. In one embodiment, the background-subtracted image may be segmented into regions of different visibility levels based on a comparison of the RGB value of each pixel with a defined color density range. In another embodiment, a color intensity map may be calculated, and clustering techniques such as k-means may be used to cluster different color groups. The RGB values ​​of the cluster centers may be used as reference points for determining the visibility level of each pixel in the corresponding cluster based on a comparison with the defined color density range. The results may include a 2D surface coverage map, which may indicate the percentage of visible surfaces (including invisible surfaces or blind spots) at different levels. The visibility level in the 2D surface coverage map corresponding to a given camera may be determined by combining the results from multiple realistic images rendered by randomization of the environment and operation.

[0033] Once the above process is repeated for each individual camera, the results can be forwarded to the global context mapper 220. The global context mapper 220 can spatially map the pixels of the individual camera's 2D surface coverage map into the 3D model of the part to create a global visible surface coverage map. For overlapping pixels (pixels viewed by multiple cameras), the highest visibility level can be considered to create the global visible surface coverage map.

[0034] Figure 4 A global visible surface coverage map 400 for a 2D image-based blind spot detector is shown. As shown, the global visible surface coverage map 400 indicates three visibility levels on a 3D model of a part. In the illustrated illustration, shading 402, 404, 406 indicates low (blind spot), medium, and high levels of visibility, respectively. Visible surface coverage can be measured by determining areas that do not include blind spots (in this case, medium and high visibility areas), such as as a fraction or percentage of the entire part surface.

[0035] Back to Figure 2 , the second modality of the blind spot detector 218 can involve a blind spot detector based on 3D geometry. In this case, the output 214 of the simulation engine 204 that defines the input of the blind spot detector 218 can include, for each individual camera, an area of ​​the part surface defined by a 3D model of the part within the FOV of the camera. Typically, the 3D model of the part can be constructed by generating a polygonal mesh that defines planar surface elements. The planar surface elements can include, for example, tessellated triangles, quadrilaterals, or other simple convex polygons. In one embodiment, the output 214 used by the blind spot detector 218 based on 3D geometry can include a collection of planar surface elements contained within the FOV of the camera for each camera. In this modality, the blind spot detector 218 does not necessarily require realistic rendering.

[0036] The 3D geometry-based blind spot detector 218 can operate by determining the visibility levels at multiple locations on the part surface based on the angle between the local part surface and the FOV plane of the camera. The FOV plane of the camera refers to the plane defined by the image acquisition surface (lens) of the camera. Therefore, the 3D geometry-based blind spot detector 218 can determine, for each camera, a measurement of the angle between the part surface and the FOV plane of the camera at multiple locations in the area within the FOV of the camera. For each location, the visibility level can be determined by comparing the corresponding angle with a defined angle range. In some embodiments, multiple angle ranges can be defined to classify multiple visibility levels.

[0037] In a simple implementation, locations for angle measurements can be defined using landmarks. Landmarks can be placed on a planar surface formed by geometric shapes on the surface of a part defined by a 3D model of the part. Landmarks can represent all points on a given planar surface. Landmarks can be specified, for example, in part information used to create the digital twin 202. This approach is computationally efficient and is particularly suitable for uncomplex part surfaces (e.g., including regular geometric shapes).

[0038] refer to Figure 5, part 504 may include several planar surfaces, such as 504a, 504b, 504c, 504d, etc., which may have landmarks placed thereon. For each individual camera 502a, 502b, 502c, the angle between the camera's FOV plane and the planar surface within the camera's FOV may be measured at the corresponding landmark point. In the schematic diagram shown, θ a represents the angle between the FOV plane of camera 502a and surface 504b, θ b represents the angle between the FOV plane of camera 502b and surface 504a, and θ c represents the angle between the FOV plane of camera 502c and surface 504a.

[0039] The angles described in this article are 3D angles. In practice, the measurement of each angle can be determined by measuring the angle between the normal of the planar surface (surface normal) and the normal of the camera FOV plane (camera normal). Therefore, visibility may be highest when the angle is 180 degrees (the planar surface faces the camera FOV plane), and visibility is almost zero when the angle drops to about 90 degrees. In the case of overlapping planar areas within the FOV of multiple cameras, the highest measured angle can be stored.

[0040] In another embodiment, instead of landmarks, multiple locations may correspond to planar surface elements defined by a polygonal mesh on a 3D model of a part, respectively. In this case, for a given camera, a surface normal may be calculated for each planar surface element within the camera's FOV. Based on this, the angle between each planar surface element and the camera's FOV plane may be measured by measuring the angle between the corresponding surface normal and the camera normal. This process may be repeated for each simulated camera. Therefore, for each planar surface element on the 3D model of the part, an angle indicating its visibility may be stored. In the case of planar surface elements within the FOV of overlapping multiple cameras, the highest measured angle may be stored. The stored angle of each planar surface element may be compared with a defined angle range to determine the visibility level of each planar surface element. This technique may be particularly suitable for part surfaces having complex shapes because it may be used to accurately determine the gradient of the planar surface angle along the part surface.

[0041] In yet another embodiment, for an area within the FOV of a corresponding camera, multiple locations may be determined by sampling points on a UV map of a part surface defined by a 3D model of the part. UV mapping is a known technique for mapping (unwrapping) a 3D mesh into a 2D image. A 2D image may be defined by UV coordinates corresponding to vertex information from a 3D mesh. This embodiment may involve generating a UV map using a 3D mesh of a part. Points may be sampled on a UV map, for example, using random or blue noise sampling. The sampled points may already include UV projection information, which includes a surface normal. For each sampled point in the FOV of a camera, an angle may be determined directly based on knowledge of the surface normal and the camera normal. This process may be repeated for each simulated camera. Thus, for each sampled point on a UV map, an angle indicating its visibility may be stored. In the case of overlapping points within the FOV of multiple cameras, the highest measured angle may be stored. The stored angle of each point may be compared with a defined angle range to determine the visibility level of each point.

[0042] The global context mapper 220 can map visibility levels for locations determined for individual cameras into the 3D model of the part to create a global visible surface coverage map. In one embodiment, the visibility level corresponding to a given camera can be determined by combining results from multiple images rendered by environmental and operational randomization.

[0043] Figure 6 An example of a global visible surface coverage map 600 for a 3D geometry-based blind spot detector is shown. As shown, the global visible surface coverage map 600 indicates three levels of visibility on a 3D model of a part. In the illustrated illustration, shadows 602, 604, 606 indicate low (blind spot), medium, and high levels of visibility, respectively. In an example implementation, the shadow 602 indicating low visibility may include points or plane surface elements with angles in the range of [0 to 100 degrees], the shadow 604 indicating medium visibility may include points or plane surface elements with angles in the range of [100 to 130 degrees], and the shadow 606 indicating high visibility may include points or plane surface elements with angles in the range of [130 to 180 degrees]. Visible surface coverage (in this case, medium and high visibility areas) may be measured by determining areas that exclude blind spots, for example as a fraction or percentage of the total part surface.

[0044] Continue to refer Figure 2, the output 222 of the surface coverage measurement engine 216 can indicate a measurement of visible surface coverage determined using the global context mapper 220. In one embodiment, the surface coverage measurement engine 216 can incorporate both a 2D image-based blind spot detector and a 3D geometry-based blind spot detector. In this case, the output 222 can include a combination of visible surface coverage determined using both modalities of blind spot detection. Doing so can define a metric for optimizing the camera configuration that takes into account the geometry of the part and the behavior of light in the physical environment of the inspection system. In an example, the output 222 can be determined by assigning appropriate weights to the results generated using the 2D image-based blind spot detector and the 3D geometry-based blind spot detector.

[0045] As a further feature according to the disclosed embodiments, the visible surface coverage determined by the surface coverage measurement engine 216 can be integrated with a measure of the AI ​​model performance to define optimization goals. To this end, the system 200 can include an AI engine 226. The AI ​​engine 226 can provide another realistic metric to quantitatively measure the robustness of the optimized camera configuration parameters.

[0046] The AI ​​engine 226 may receive the realistic image 224 as input from the simulation engine 204. In this case, the simulation engine 204 may be combined with the defect generator 210 to create defect data in the realistic image 224. The defect generator 210 may be configured to extract defects on high-risk areas of the part. Such high-risk areas may be defined, for example, in the part information used to create the digital twin 202. The defect generator may randomize the defects and apply the defects to the 3D model of the part based on the realistic image that can be rendered by executing the camera simulator 206 and the environment simulator 208.

[0047] The realistic image 224 received by the AI ​​engine 226 may include both defective images and non-defective images. Defective images may be automatically labeled, for example, by locating defects with a bounding box. The realistic image 224 may be used to create a labeled data set 228. In some embodiments, the size of the data set 228 may be increased by using data augmentation techniques. A model trainer 230 may be executed to train an AI model for defect detection using the data set 228. The AI ​​model may include, for example, a deep neural network. Training may involve a supervised learning process based on defect labels. A model performance evaluator 232 may be used to test the trained AI model and measure its performance. The measurement of performance may include the accuracy or success rate of the trained AI model correctly predicting defects in unlabeled input images during the evaluation period. The output 234 of the AI ​​engine 226 indicating the measured performance of the trained AI model may be incorporated into the optimization target together with the output 222 of the surface coverage measurement engine 216.

[0048] The optimization engine 236 may be used to determine an optimized configuration for the image acquisition system by generating an updated configuration in each iteration based on an evaluation of an optimization goal. The optimization goal may include an objective function defined based on a measurement of visible surface coverage. However, the optimization goal may be open ended to incorporate other metrics as desired. For example, according to the disclosed embodiments, an objective function may be defined based on a combination of a measurement of visible surface coverage and a measurement of the performance of an AI model trained using rendered synthetic images.

[0049] The optimization engine 236 may include a goal evaluator 238 to evaluate the optimization goal at each iteration. According to the disclosed embodiment, the optimization goal can be evaluated based on the outputs 222 and 234 of the surface coverage measurement engine 216 and the AI ​​engine 226, respectively. The optimization goal can be evaluated by assigning corresponding weights to the measured visible surface coverage and the measured AI model performance. For example, in one embodiment, the optimization goal can be evaluated as a weighted sum of the outputs 222 and 234. Depending on the weight, the optimization may be biased towards achieving higher visible surface coverage or higher AI model performance.

[0050] The optimization engine 236 may also include a configuration generator 240 to generate an updated configuration of the image acquisition system based on the evaluation of the optimization goal. The configuration generator 240 may receive the current value of the optimizable parameter 242 from the simulation engine 204 and the evaluated optimization goal from the goal evaluator 238 as input to determine the updated value of the optimizable parameter 246. To determine the updated configuration, the configuration generator 240 may be informed of the constraints and search space 244 for each parameter to be optimized. The constraints and search space 244 may be obtained from the digital twin 202. For example, Figure 1 As shown, if the camera position is the parameter to be optimized, the search space can be restricted to a fixed length horizontally. In addition, the cameras may not be placed on top of each other, so that each camera must be constrained to have a unique position. Similarly, if the camera angle is the parameter to be optimized, the search space can be restricted to [-180 to 180 degrees].

[0051] In one suitable embodiment, the optimization engine 236 may include a genetic algorithm that may generate a new camera configuration population, for example, by performing selection crossover and mutation in the direction of maximizing the optimization objective. Other suitable embodiments of the optimization engine 236 may include a reinforcement learning (RL) algorithm, a particle swarm optimization (PSO) algorithm, and the like.

[0052] The updated values ​​of the optimizable parameters 246 may be transmitted back to the simulation engine 204 to begin the next iteration of the above steps. Iterations may be performed until a convergence criterion is met. For example, the convergence criterion may be met when the evaluated optimization objective has reached a predetermined threshold. Upon achieving the convergence criterion, the final values ​​of the optimizable parameters may be stored as a final configuration 248 of the image acquisition system to be used in the deployment phase.

[0053] Figure 7 The disclosed method is shown deployed on a vision-based inspection system according to one embodiment. As shown, the vision-based inspection system 700 may include an image acquisition system 702, which includes multiple cameras that can be triggered to capture images of various parts on an end-of-line conveyor system 704. The images captured by the image acquisition system 702 may be processed by a defect detector 704, which may include a computing system that may use an AI model and / or a computer vision algorithm to detect the presence of defects in the part.

[0054] The inspection system 700 may be capable of visually inspecting different types of parts that may have different nominal geometries. The database 706 may store a corresponding optimized configuration of the image acquisition system for each type of part. In some embodiments, the database 706 may also store a corresponding trained AI model specific to each type of part (e.g., generated by the AI ​​engine 226). When a batch of a given type of part is introduced on the production line, the part ID may be specified, for example, by an operator or controller, to retrieve the corresponding camera configuration and AI model from the database 706. The retrieved camera configuration may be used to configure the image acquisition system 702. The retrieved AI model may be deployed to the defect detector 704.

[0055] In an alternative embodiment, the optimization of the camera configuration can be generic for different types of parts. In this case, the above method can be implemented by using 3D models of different parts with different nominal geometries at each iteration, including simulation, surface coverage measurement (and in some embodiments, AI model performance measurement), and optimization. From this, a final generic configuration of the image acquisition system can be determined for vision-based inspection of different parts. This can avoid the need to change the camera configuration each time a new batch of parts arrives at the inspection system.

[0056] Figure 8An example of a computing system 800 that can support visual quality inspection of manufactured parts on a shop floor in accordance with the disclosed embodiments is shown. In an example, the computing system 800 can be configured as a powerful multi-GPU workstation among other types of computing devices. The computing system 800 includes at least one processor 810 that can take the form of a single or multiple processors. The processor 810 may include one or more CPUs, GPUs, microprocessors, or any hardware device suitable for executing instructions stored on a memory including a machine-readable medium. The computing system 800 also includes a machine-readable medium 820. The machine-readable medium 820 may take the form of any non-transitory electronic, magnetic, optical, or other physical storage device that stores executable instructions, such as Figure 8 Shown are simulation instructions 822, surface coverage measurement instructions 824, and optimization instructions 826. Thus, the machine-readable medium 820 may be, for example, a random access memory (RAM) such as a dynamic RAM (DRAM) flash memory, a spin transfer torque memory, an electrically erasable programmable read-only memory (EEPROM), a storage drive, an optical disk, or the like.

[0057] The computing system 800 can execute instructions stored on the machine-readable medium 820 through the processor 810. Executing the instructions (e.g., simulation instructions 822, surface coverage measurement instructions 824, and optimization instructions 826) can cause the computing system 800 to perform any of the technical features described herein, including performing according to any of the simulation engine 204, surface coverage measurement engine 216, and optimization engine 236 features described above.

[0058] The above-described systems, methods, devices, and logic (including the simulation engine 204, the surface coverage measurement engine 216, and the optimization engine 236) can be implemented in many different ways in many different combinations of hardware logic circuits and executable instructions stored on machine-readable media. For example, these engines may include circuits in a controller, a microprocessor, or an application-specific integrated circuit (ASIC), or may be implemented with a combination of discrete logic or components, or other types of analog or digital circuits combined on a single integrated circuit, or distributed between multiple integrated circuits. Products such as computer program products may include a storage medium and machine-readable instructions stored on the medium, which, when executed in an endpoint, a computer system, or other device, cause the device to perform operations according to any of the above descriptions, including any features according to the simulation engine 204, the surface coverage measurement engine 216, and the optimization engine 236. The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, and / or a wireless network.

[0059] The processing capabilities of the systems, devices, and engines described herein (including the simulation engine 204, the surface coverage measurement engine 216, and the optimization engine 236) can be distributed among multiple system components, such as distributed among multiple processors and memories that optionally include multiple distributed processing systems or cloud / network elements. Parameters, databases, and other data structures can be stored and managed separately, can be merged into a single memory or database, can be logically and physically organized in many different ways, and can be implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs can be parts of a single program (e.g., subroutines), separate programs, distributed across several memories and processors, or implemented in many different ways, such as in a library (e.g., a shared library).

[0060] Although the present disclosure has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the present design may be implemented by those skilled in the art without departing from the scope of the patent claims.

Claims

1. A computer-implemented method for configuring an image acquisition system including one or more cameras for performing vision-based inspection of parts on a production line, the method comprising: In multiple iterations, execute: executing a simulation engine for rendering a composite image of the part on the production line acquired by the one or more cameras based on the 3D model of the part and the configuration of the image acquisition system defined by the optimizable parameters, running a surface coverage measurement engine for measuring blind spots on a surface of a part for individual cameras of the one or more cameras using output of the simulation engine and thereby determining a measurement of visible surface coverage on the 3D model of the part, and executing an optimization engine for generating an updated configuration of the image acquisition system by updating the optimizable parameters based on an evaluation of an optimization goal defined by the measured visible surface coverage, The final configuration of the image acquisition system is thereby determined for use in vision-based inspection on the production line.

2. The method according to claim 1, wherein: The simulation engine is run to render a plurality of composite images for each individual camera based on randomization of environmental and operating parameters associated with the production line.

3. The method according to claim 2, wherein: The composite image rendered by the simulation engine comprises a realistic image.

4. The method according to claim 3, wherein: The output of the simulation engine used by the surface coverage measurement engine includes the realistic image, the visible surface coverage being measured by: for individual cameras, segmenting the corresponding rendered photorealistic images based on defined ranges of color density levels to generate a 2D surface overlay that delineates visible areas from shadowed areas corresponding to blind spots, Pixels of the 2D surface coverage maps for individual cameras are spatially mapped into the 3D model of the part to create a globally visible surface coverage map.

5. The method according to any one of claims 1 to 4, wherein: The output of the simulation engine used by the surface coverage measurement engine includes, for each individual camera, the area of ​​the part surface defined by the 3D model of the part that is within the field of view (FOV) of the camera, the visible surface coverage camera measures by: determining, for each individual camera, a measure of the angle between the surface of the part and the plane of the FOV of the camera at a plurality of locations in a region within the FOV of the camera, determining the visibility of each position by comparing the corresponding said angle with a defined angular range, and The visibility of the locations determined for the individual cameras is mapped into the 3D model of the part to create a global visible surface coverage.

6. The method according to claim 5, wherein: The plurality of locations respectively correspond to planar surface elements defined by a polygonal mesh of the 3D model of the part, the planar surface elements being contained in a region within the FOV of a corresponding camera.

7. The method according to claim 5, wherein: The plurality of positions are determined by sampling points on a UV map of the part surface defined by the 3D model of the part for the region within the FOV of the corresponding camera.

8. The method according to claim 5, wherein: The plurality of locations include, for an area within the FOV of a corresponding camera, landmarks on a planar surface, the landmarks being formed by geometric shapes on a surface of the part, the geometric shapes being defined by the 3D model of the part.

9. The method according to any one of claims 3 to 8, further comprising: running a defect generator to create defect data in the photorealistic image, creating a data set using said photorealistic images including said defect data, using the dataset to train an artificial intelligence (AI) model for defect detection, and Measure the performance of trained AI models, Therein, the optimization objective is defined by a combination of measured visible surface coverage and measured performance of the AI ​​model.

10. The method according to any one of claims 1 to 9, wherein: The iterations are performed until the evaluation of the optimization objective reaches a predefined threshold.

11. The method according to any one of claims 1 to 10, wherein: The optimization engine includes a genetic algorithm.

12. The method of any one of claims 1 to 11, comprising performing each of the iterations using 3D models of different parts having different nominal geometries to determine a final common configuration of the image acquisition system for inspecting the different parts.

13. The method according to any one of claims 1 to 12, wherein: The optimizable parameters are selected from the group consisting of: position, angle, field of view, exposure, color mode, analog gain, and depth of focus of the one or more cameras.

14. A non-transitory computer-readable storage medium comprising instructions which, when processed by a computing system, configure the computing system to perform the method according to any one of claims 1 to 13.

15. A system for configuring an image acquisition system including one or more cameras for performing vision-based inspection of parts on a production line, the system comprising: one or more processors, and A memory storing an algorithm module executable by the one or more processors, wherein the algorithm module comprises: a simulation engine configured to render a composite image of the part on the production line acquired by the one or more cameras based on the 3D model of the part and the configuration of the image acquisition system defined by the optimizable parameters; a surface coverage measurement engine configured to use the output of the simulation engine to measure blind spots on a surface of a part for individual cameras of the one or more cameras and thereby determine a measurement of visible surface coverage on the 3D model of the part; and an optimization engine configured to generate an updated configuration of the image acquisition system by updating the optimizable parameters based on an evaluation of an optimization goal defined by the measured visible surface coverage, Wherein, the simulation engine, the surface coverage measurement engine, and the optimization engine are capable of executing in multiple iterations to determine a final configuration of the image acquisition system for vision-based inspection on the production line.

Images