Adaptation of illumination settings for optical measurement and inspection systems

Abstract

The invention relates to adaptation of illumination settings for optical measurement and inspection systems. The invention relates to a method of optically inspecting a feature of interest on a workpiece. The method includes: simulating irradiation of a workpiece model; training an AI agent or ML model based on a composite image obtained from the illumination to automatically control the plurality of light sources; inferring a characteristic of the feature of interest in the image captured by the camera; the trained AI agent controls a plurality of light sources to select a light setting that illuminates a feature of interest; the illuminated feature of interest is optically examined while illuminating the feature of interest with the selected light settings. Illuminating includes: providing 3D data of the workpiece model; approaching a representation of the feature on the workpiece model with a virtual camera and a plurality of virtual light sources arranged corresponding to the arrangement of real light sources; rendering a plurality of base images imaging a representation of the feature from a viewpoint of the virtual camera under a set of specific predefined light settings of the virtual light source; a composite image of all possible light settings is obtained by compositing the base image.

Description

Field of the Invention

[0001] The present invention generally relates to the field of optical measurement and inspection systems and to optically measuring and inspecting workpieces. More specifically, the present invention relates to the automatic adaptation of illumination settings of an optical measurement and inspection system for improved optical measurement or inspection of workpieces. Background Art

[0002] Optical measuring machines and optical inspection machines are used in various applications in quality management, quality assurance, and production. Compared to tactile measurement techniques, optical measurement and inspection processes use lights for non-contact measurement or inspection of the characteristics of a test object. This technique utilizes the physical principles of absorption and reflection to capture the characteristics of an entire surface rather than just individual measurement points.

[0003] Optical quality control involves the visual inspection of workpieces or their geometric features using cameras. The acquired images are used to detect defects or deviations from nominal dimensions and / or shapes visually, programmatically, or algorithmically (e.g., using image processing algorithms). The quality and usability of such recorded images for quality control highly depend on lighting. Optical inspection machines and / or measuring machines typically include a number of lighting elements that can be controlled independently of each other, including different absolute or relative illumination levels of multiple lighting elements. These lighting elements can include a top lighting element, a rear lighting element, and a front lighting element, where the front lighting typically includes an annular light arranged around a camera for inspecting a workpiece. The top lighting can include a coaxial LED ceiling light, and the backlighting can include a telecentric LED backlight. Both the top lighting and the backlighting can be fixedly mounted in the optical inspection machine and / or measuring machine. The front lighting can be movable relative to the object to be measured and / or inspected (e.g., together with the camera).

[0004] Accordingly, several top light configurations, rear light configurations, and annular light configurations can be selected according to a given object (i.e., its geometric shape with holes and edges and its specific surface characteristics). When selecting a specific lighting configuration, certain boundary conditions must also be considered. These boundary conditions include the orientation and distance of one or more cameras, lighting and object characteristics, and the settings of the cameras, as well as stray light from the surrounding environment.

[0005] Based on part geometry, material properties, etc., various light settings will cause various image effects, such as shadows or reflections. Currently, for the purpose of visual inspection, a manual step-by-step tuning process is required to empirically converge to a satisfactory light setting. Therefore, the expertise of the operator is indispensable, making the entire cumbersome process non-scalable. Summary of the Invention

[0006] Accordingly, the object of the present invention is to provide an improved method and an improved system which allow finding the optimal illumination settings of an optical measuring machine.

[0007] A particular object is to provide such a method and system which allow autonomously finding the optimal illumination settings.

[0008] A first aspect of the present invention relates to a method for optically inspecting a feature of interest on a workpiece using an optical inspection and / or measurement system comprising at least one camera and a plurality of light sources, the light sources allowing selection of a plurality of different light settings. The method comprises:

[0009] - inferring properties of the geometric feature of interest on the workpiece in at least one image captured by at least one camera;

[0010] - controlling the plurality of light sources based at least on the inferred properties to select a light setting for illuminating the feature of interest; and

[0011] - optically inspecting the illuminated feature of interest using at least one camera when the feature of interest is illuminated with the selected light setting.

[0012] According to this aspect of the present invention, the method further comprises simulating the illumination of a workpiece model and training an AI model at least partly based on the simulated illumination to automatically control the plurality of light sources. Simulating the illumination of the workpiece model comprises:

[0013] - providing 3D data of the workpiece model;

[0014] - approaching a representation of the feature on the workpiece model with at least one virtual camera and a plurality of virtual light sources, wherein the virtual light sources are arranged to correspond to the arrangement of the light sources of the optical measurement system and allow selection of the same plurality of light settings as the light sources of the optical measurement system;

[0015] - rendering a plurality of base images imaging the representation of the feature from the viewpoint of the virtual camera under a specific predefined set of light settings of the virtual light sources; and

[0016] - obtaining a composite image of all possible light settings of the virtual light sources by synthesizing the base images.

[0017] According to this aspect of the present invention, training the AI model is based on the obtained composite images and the trained AI model automatically performs controlling the plurality of light sources.

[0018] According to some embodiments, the method comprises defining an image quality key performance indicator which measures the adequacy of the image of the feature to be used for optically inspecting the feature. For example, defining the image quality key performance indicator may comprise automatically deriving the image quality key performance indicator from the image.

[0019] According to some embodiments, training the AI model includes a reinforcement learning loop for training a reinforcement learning agent (e.g., one of a deep Q-network, proximal policy optimization, Advantage Actor-Critic, and Actor-Critic with Experience Replay (ACER)). For example, the reinforcement learning agent can be retrained fully or semi-automatically while controlling multiple light sources.

[0020] According to some embodiments, defining the image quality key performance indicator includes automatically deriving the image quality key performance indicator from the image and includes at least one of the following:

[0021] - Considering the brightness and / or contrast around the feature of interest;

[0022] - Measuring the density and / or deviation of the contour points detected using an image processing algorithm around a specific region, curve, or polygon; and

[0023] - Measuring the complexity of the contour detected using an image processing algorithm.

[0024] According to some embodiments, single-shot parameter estimation is applied in a supervised machine learning framework to directly estimate the optimal light setting from the input image obtained using a predefined light setting. For example, the AI model can be a classifier or a regressor, e.g., having a Deep Neural Network (DNN) architecture.

[0025] According to some embodiments, simulating the illumination of the workpiece model includes simulating the illumination of multiple different workpiece models having multiple different geometric features, where the different geometric features differ at least in terms of their shape and at least one of material, color, roughness, and specular reflection.

[0026] According to some embodiments, the method includes moving a probe of an optical measurement system closer to the feature of interest, the probe including at least one camera and at least a first subset of the multiple light sources, where controlling the multiple light sources includes controlling the light sources of the first subset. For example, the probe can be moved closer to the feature of interest to infer the characteristics of the feature of interest and / or optically inspect the feature of interest. Moreover, controlling the multiple light sources can optionally include controlling (the light sources of the second subset) that are not included in the probe, can be controlled separately, and are immovable relative to the workpiece.

[0027] According to some embodiments, a feature of interest is approached by a probe for optical inspection of the feature of interest, wherein training the AI model includes training to automatically control the movement of the probe relative to the feature of interest to vary the distance and angle of the probe relative to the feature of interest, and controlling the plurality of light sources includes automatically controlling the movement of the probe relative to the feature of interest by the trained AI model.

[0028] According to some embodiments, a first subset of the light sources is arranged around at least one camera in one or more concentric rings (e.g., in at least three concentric rings).

[0029] According to some embodiments, the plurality of light sources are arranged as at least eight (e.g., twelve or more) individually controllable light groups.

[0030] A second aspect of the present invention relates to an optical measurement system for optical inspection of a feature of interest on a workpiece, the optical measurement system including a training unit and one or more optical measurement devices. Each (or said) optical measurement device includes a computing unit, at least one camera, and a plurality of light sources, the light sources allowing selection of a plurality of different light settings, and the computing unit being configured to control the optical inspection function of the optical measurement system for optical inspection of the feature of interest. The training unit is configured to simulate illumination of a workpiece model to obtain synthetic images, train an AI model at least in part based on the obtained synthetic images to automatically control the plurality of light sources, and provide the trained AI model to the one or more optical measurement devices. Simulating the illumination of the workpiece model includes:

[0031] - providing 3D data of the workpiece model;

[0032] - approaching a representation of the feature on the workpiece model with a virtual camera and a plurality of virtual light sources, wherein the virtual light sources are arranged to correspond to the arrangement of the light sources of the one or more optical measurement devices and allow selection of the same plurality of light settings;

[0033] - rendering a plurality of base images that image a representation of the feature from the viewpoint of the virtual camera under a specific predefined set of light settings of the virtual light sources; and

[0034] - obtaining synthetic images of all possible light settings of the virtual light sources by synthesizing the base images.

[0035] Each (or said) optical measurement device is configured to:

[0036] - infer characteristics of a feature of interest on the workpiece in at least one image captured by at least one camera;

[0037] - automatically control the plurality of light sources to select a light setting for illuminating the feature of interest by the trained AI model and at least based on the inferred characteristics; and

[0038] - Optically inspect the irradiated feature of interest using at least one camera while irradiating the feature of interest with the selected light setting.

[0039] According to some embodiments of the optical measurement system, at least one of the one or more optical measurement devices includes a probe that is movable relative to the workpiece to approach the feature of interest, the probe including at least one camera and at least a first subset of a plurality of light sources, wherein controlling the plurality of light sources includes controlling the light sources of the first subset. For example, the feature of interest can be approached by the probe for inferring the characteristics of the feature of interest and / or for optically inspecting the feature of interest. Moreover, one or more of the optical measurement devices can include a second subset of light sources not included in the probe, which can be controlled separately and are immovable relative to the workpiece. In such a case, controlling the plurality of light sources can also include controlling the light sources of the second subset.

[0040] According to some embodiments of the optical measurement system, the first subset of light sources is arranged on the probe around at least one camera in one or more concentric rings (e.g., in at least three concentric rings). Moreover, the plurality of light sources can be arranged in at least eight (e.g., at least twelve) individually controllable light groups.

[0041] According to some embodiments, the optical measurement system is configured to perform the method according to the first aspect of the present invention. The AI model can be a reinforcement learning agent (such as a Deep Q-Network) or a supervised machine learning agent (such as a Deep Neural Network (DNN)).

[0042] A third aspect of the present invention relates to a computer program product including program code having computer-executable instructions for performing the method according to the first aspect of the present invention, especially when running in the system according to the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be described in detail below by reference to exemplary embodiments accompanied by drawings, wherein:

[0044] Figure 1 Illustrates the optical inspection of a workpiece by means of an optical probe of an optical measuring machine;

[0045] Figure 2 Illustrates an exemplary embodiment of a probe of an optical measuring machine including an annular lighting element having a plurality of LEDs;

[0046] Figures 3a to 3c Illustrates images of the same circular feature of a workpiece acquired with three different lighting devices;

[0047] Figure 4 Illustrates a flowchart explaining an exemplary embodiment of the method according to the present invention;

[0048] Figure 5a Shows the simulation of 3D workpieces, cameras, and lighting devices within a simulation engine for rendering synthetic images;

[0049] Figure 5b Shows synthetic images of the same circular feature of a rendered workpiece model obtained using three different lighting devices;

[0050] Figure 6 Shows a flowchart illustrating an exemplary process of using a workpiece and camera and lighting models to render a synthetic image, as Figure 4 part of the method;

[0051] Figure 7 Shows a flowchart illustrating an exemplary reinforcement learning loop for training an AI agent as Figure 4 part of the method;

[0052] Figure 8 Shows an illustrative embodiment of a system in which a deep neural network is used to directly infer the optimal light settings from image features; and

[0053] Figure 9 Shows an illustrative embodiment of an optical measurement system according to the present invention. Detailed Description

[0054] Figure 1 Shows an example of a workpiece 5 that includes a geometric feature of interest 50 optically inspected by an optical probe 1 of an optical measuring machine. The probe 1 includes at least one camera and an illuminator for illuminating the workpiece 5 and the feature of interest 50 during optical inspection. The probe 1 can be moved relative to the workpiece 5 to approach the feature of interest 50. Preferably, the probe 1 can be moved in six degrees of freedom (6DOF) to allow inspection of the feature at different distances and angles.

[0055] Figure 2 Shows an example of a probe 1 used in an optical measuring machine. A camera 10 is disposed at the center of the probe 1 and is surrounded by three concentric rings of light sources 11 (e.g., light-emitting diodes (LEDs)). Each ring is divided into four sectors such that, in this particular setup, the intensity levels of twelve annular light sectors can be individually controlled and tuned. Additionally, the optical measuring machine can include additional light sources, such as top light and back light (not shown here), that can also be individually controlled and tuned.

[0056] Of course, other light-setting geometries than concentric rings are possible. Also, various light source types or combinations thereof can be used, such as different LED types, discharge or fluorescent lamps, and halogen lamps. The light source can provide light with a uniform wavelength, such as white light, or provide electromagnetic radiation (light) of different wavelengths, i.e., colors (RGB) and / or infrared (IR) or ultraviolet (UV) radiation.

[0057] Another parameter that can be optimized is exposure. Exposure is basically controlled by the lens aperture and the duration for which the lamp exposes the sensor. Although the exposure can be fixed to one level (i.e., E = 1) by using only one aperture and shutter configuration of the camera, optionally, various exposure levels can be added to the variables to be tuned together with the intensity quantization level of the lamp using reinforcement learning.

[0058] If N is the number of tunable light sets, E is the number of possible exposure levels, and Q is the number of quantization levels of the luminous intensity, then there are T=(E×Q) N total possible light settings, which increases exponentially with N. Assuming a setting where N = 14 (i.e., twelve annular sectors and two additional lamps for the top lamp and the back lamp respectively), E = 1 and the light intensity can be set between 0% and 100% in 10% steps (i.e., Q = 11), T is approximately 380 trillion, which makes it difficult to exhaustively test all solutions. Therefore, the present invention proposes to automatically infer the optimal light setting so as to provide sufficient images for subsequent optical quality control without the need for manual intervention.

[0059] Figures 3a to 3c Three images of the same circular feature of a workpiece obtained with different light settings are shown. Typical quality inspections in manufacturing include measuring the diameter of a circular drill hole and verifying that the dimension falls within the tolerance limits. Therefore, if an image processing tool (such as an algorithm based on contour detection) is used to detect and fit a circle, the light setting must be chosen to optimize the image quality in the region of the geometric feature of interest, i.e., to optimize the brightness and contrast.

[0060] An AI-based solution is proposed that implicitly utilizes object features such as geometric shapes, material properties, etc. The AI-based solution described herein includes a reinforcement learning (RL) framework: in reinforcement learning, an agent learns by interacting with the environment through trial and error and receiving positive or negative rewards as feedback. The agent is an intelligent decision maker. The environment receives action signals from the agent and returns observation signals to the agent. The present invention is not limited to RL per se, but also allows the use of implementations of supervised machine learning (SML), particularly neural networks applying deep learning. In the method using RL, the agent is trained to explore the space of light settings according to the key performance indicator (KPI) of image quality to maximize the total reward.

[0061] Figure 4 Steps of an exemplary embodiment of a method 100 for optical inspection of a workpiece according to the present invention are shown. In a first step of the method 100, the image quality KPI is defined 110 as a score, for example. The KPI measures the adequacy of the image to be used for optical quality control. The image quality score is used to guide the entire AI-based method until convergence to an acceptable value. The KPI can be automatically derived from the image in several ways. For example, the score can:

[0062] - Consider the brightness and / or contrast in the region of interest around the geometric feature to be measured;

[0063] - Measure the density and / or deviation of the contour points detected with an image processing algorithm (such as PC-DMIS of Hexagon Manufacturing Intelligence) around a specific region, curve, or polygon;

[0064] - Measure the complexity of the contour detected with an image processing algorithm (for example, assuming that the geometric feature being measured exhibits low complexity or entropy), where the contour complexity can be calculated, for example, by line segment approximation, such as using the Ramer-Douglas-Peucker algorithm, and then calculating the number of turning points normalized by the length of the contour line; and / or

[0065] - Consist of an adjoint and / or metrology-specific definition of the KPI.

[0066] In the next step, the simulation engine simulates the 120 workpiece models under various lighting conditions. By rendering multiple synthetic images, the simulation engine enables the generation of a scalable training dataset for the RL model. By varying the workpieces and geometric features of interest to cover various shapes in the simulation engine, as well as their materials, colors, roughness, specular reflection, etc., a large example corpus can be created. For this purpose, the light settings as well as the camera must be modeled within the simulation engine. This is described in more detail below with respect to Figure 5a and Figure 5b described in more detail.

[0067] Then, a reinforcement learning loop (RL loop) is used to train the 130 AI agent. This is described in more detail below with respect to Figure 7 Alternatively, based on the dataset of 120, a machine learning (ML) model such as a deep neural network can be trained for 130, as described in more detail in Figure 8 Optionally, training the AI agent or the ML model can be partially based on real data (e.g., real images of real workpieces), e.g., for fine-tuning.

[0068] When the AI agent or the ML model has been trained, the optical measuring machine can use the AI agent or the ML model to inspect actual workpieces.

[0069] For a given geometric feature of interest on a workpiece that needs to be inspected by the camera of the optical measuring machine, this includes automatically adapting the material, color, roughness, specular reflection, etc. of the feature, as well as adapting to the surrounding conditions (e.g., stray light) and the conditions of the camera device 140.

[0070] The AI agent or the ML model automatically determines the optimal lighting for the determined feature under the detected conditions and accordingly controls the lighting of the 150 optical measuring machine. Then, the optical measuring machine uses the lighting selected by the AI / ML to optically inspect the feature of interest on the 160 workpiece. The AI agent or the ML model can also be continuously retrained and improved on real data, e.g., involving a human operator in the loop to replace and / or supplement the KPI-based reward mechanism and / or add new datasets to supervised retraining.

[0071] Figure 5a and Figure 5b shows the use of the simulation engine to render synthetic images of workpieces as the basis for AI training ( Figure 4 step 120 of the method). Figure 5a shows the simulation 4 in the 3D rendering software "Blender", and Figure 5bShows three images 40, 40', 40' rendered using the simulation under different light settings. The representation 5' of the workpiece in the example shown is a 3D metal block, which includes representations of various geometric features, including a representation 50' of a circular hole on the top of the workpiece as a feature of interest (indicated here by a white circle). The virtual light source 11' is arranged as Figure 2 shown, that is, arranged as three concentric circles (indicated here by dashed lines). Each light source of the lighting device is modeled individually and then grouped together into controllable sectors. In addition, the camera installed in the optical measurement device is modeled.

[0072] The present invention relies on rendering synthetic images 40, 40', 40' in a simulation engine, which enables the generation of a scalable training dataset for the RL or SML models defined above. By changing the workpiece and features of interest to cover various shapes in the simulation engine as well as their materials, colors, roughness, specular reflection, etc., a large example corpus can be created. For this purpose, the light setup as well as the camera must be modeled within the simulation engine.

[0073] Figure 6 is a flowchart showing an exemplary process for generating synthetic images to utilize the simulation environment. All process steps can be scripted. First, a 3D object 122 having geometric features (features of interest) to be measured is provided. This means that the 3D object is imported, for example, as a CAD model, or made manually directly within the simulation engine.

[0074] Various parameters of the object can be randomized 123. This tuning of the object properties (such as material, color, roughness, specular reflection, etc.) within the engine allows for data augmentation or diversification.

[0075] Then the virtual camera and the light setup can be moved 124 to visually inspect any specific location on the virtual 3D object. For any configuration of the object properties and positions defined above, a set of base images is rendered 126 for a set of specific predefined light settings. Assuming linearity of the sensor of the imaging system, an image of any lighting pattern can be obtained 128 by synthesizing the base images (i.e., the superposition of the base images). Synthesis is a linear operation and the image corresponding to a given light setting can be calculated faster from the base images than rendering them from scratch. Optionally, the simulation engine can utilize the GPU to render the base images faster.

[0076] Figure 7 Shows a reinforcement learning loop (RL loop 30) of an exemplary method according to the present invention, in which the AI agent is trained ( Figure 4Step 130 of the method recommends appropriate actions, where the actions are adjustments to the light intensity within the lighting device. In the RL loop example shown, the lighting device includes eight tunable lamp sectors 45 (composed of six annular lamp sectors, a top lamp, and a bottom lamp), and the actions are 5% step increases or decreases in the intensity of each lamp sector. Thus, the AI agent can take sixteen different actions.

[0077] The agent and the environment continuously interact with each other. At each time step, the agent selects an action for the environment based on its policy. The policy is a mapping from states to probability distributions of actions. The selected action is applied to the environment. During training, the policy balances exploitation, so it acts on what has been learned, and exploration by trying random actions.

[0078] The environment returns an observation. This observation is a partial description of the world state. It is not complete because it does not include all relevant information affecting the light settings, such as temperature or material properties.

[0079] In the RL loop 30 shown, several reinforcement learning algorithms can be used, such as Deep Q-Network (DQN), Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), or Actor-Critic with Experience Replay.

[0080] The environment is the optical inspection system 1 and the workpiece 5 when applying the agent to a real optical measurement system, or the virtual light source 11’ and the virtual workpiece 5’ when applying the agent to a simulation. The image 40 shows the observation of the system at a first time point (l t ) e.g., a synthetic image rendered by a simulation engine, as described above regarding Figure 5a The agent selects an action according to the policy 34, i.e., increasing or decreasing the intensity of one of the tunable lamps or lamp sectors. The selected action is applied 36 to the current light settings of the lamp sector 45, resulting in receiving an observation result 40’ at a second time point (l t+1 ).

[0081] Then, it is determined 38 whether the KPI of the image 40’ using the selected lighting 45 meets or exceeds a predetermined threshold, i.e., whether the image quality is sufficient. If the KPI does not reach the threshold, the agent is penalized and continues the loop 30, now using the image obtained in the previous iteration 40’ as the input for the next iteration. This process is repeated until the KPI meets or exceeds the predefined threshold, then a large positive reward is granted, and the loop is abandoned.

[0082] The agent learns to perform in an environment by repeatedly experimenting over multiple such application scenarios (episodes). Over time, the agent learns to take actions that maximize its expected reward. The policy is updated based on the agent's past experience and rewards.

[0083] Optionally, additional parameters beyond the light intensity level can be optimized. These additional parameters can include the pose of the lighting device relative to the geometric feature of interest. For example, the distance and angle of the measurement device (i.e., the optical probe) relative to the workpiece can be adapted. Additionally, in some embodiments, the angle and / or position of one or more light sources can be adapted with respect to other hardware (i.e., cameras and / or other light sources) by tilting and / or shifting. The additional parameters can also include the on / off frequency of the light source or the light spectrum (e.g., RGB colors, IR, UV), as well as the exposure level of the camera and other imaging parameters.

[0084] The additional parameters can also include effects due to heating or cooling of the illumination lamp, i.e., thermal-induced air turbulence or expansion of the illumination system (e.g., of the optical probe). The additional parameters can also include the position or time of measurement, e.g., due to position-specific or measurement-time-specific stray light. Basically, any system characteristic that shows significant tool-to-tool variation or accompanies a location / customer-specific individual configuration that may not have been covered in standard training can be an additional parameter, e.g., any effect that causes under- or overexposure of the camera image.

[0085] Advantageously, using this RL framework, annotations from human annotators are not required. In contrast to the SML framework, there is no need to manually define and record the optimal light settings for a large corpus of workpieces and their features, which would be a tedious task for an expert. Instead, the search for the optimal light settings is only driven by the KPIs defined above. Due to the parameter space exploration enabled by RL, the proposed algorithm is able to adapt to conditions unseen during training, thus minimizing the risk of getting stuck in suboptimal solutions. Additionally, the method does not rely on handcrafted features that describe geometry, materials, etc., but simply uses image features as input, which can implicitly contain such relevant information about object features.

[0086] The RL agent can be trained on experimental data and / or synthetic data that can be obtained in the laboratory. The same synthetic idea can also be used when training on real data, such that only a small number of base images need to be acquired for any given workpiece and / or its geometric features. Transfer learning can be applied to fine-tune the model to a specific scenario or environment. Finally, inference can be performed by applying the optimal learned strategy to any new workpiece or feature, regardless of their geometry, material, etc.

[0087] Optionally, the trained agent can directly access the programs that control the light settings and act directly on them following the optimal policy.

[0088] Optionally, a set of base images can first be obtained. Then, the adjustments to the light settings suggested by the optimal policy are reflected as a synthesis of these base images. Finally, the optimal image for visual inspection can also be a synthesized image obtained as the optimal combination of the base images.

[0089] Alternative solutions rely on a supervised machine learning (SML) framework rather than the RL framework described above. Figure 8 An example of a single light setting estimation using such a framework is shown, for example, based on a deep neural network (DNN) 33 as an AI model for classification or regression of light intensity. In this context, single parameter estimation can be applied to directly estimate the optimal lighting parameters 45 from the input image 40 obtained using predefined lighting conditions (i.e., predefined default settings of the lights). As in the RL-based method, this alternative method will also not require the design of handcrafted features that describe object geometry, material, etc., since deep neural networks (e.g., convolutional neural networks (CNNs) or transformer networks) will learn to extract the relevant image features for the task of light setting prediction.

[0090] For example, based on the input image 40, an SML model (e.g., using a deep neural network (DNN) 33 as an AI model) learns the features related to the optimization of the predicted light intensity. Then, an N-dimensional vector representing the predicted intensity of each light in the setting can be output, which allows the estimation of the optimal lighting parameters 45. Alternatively, any classical machine learning algorithm for classification and / or regression can be used instead of the DNN.

[0091] Contrary to the RL method, this SML method will not require the definition of image quality KPIs. However, being supervised, this alternative method will need to rely on a large image corpus and its corresponding manually labeled optimal light settings selected by experts.

[0092] Compared with the RL method, the SML method has several drawbacks, such as cumbersome manual annotation. For each workpiece and / or its features in the training database, experts will need to tune the light settings to obtain the optimal solution and then record this final setting as the ground truth. Similarly, experts will also need to tune the light settings in the simulation engine to annotate the synthesized images. Moreover, it is difficult to define a general predefined default light setting to obtain the input images, and there is a risk of finding suboptimal solutions, especially for special conditions not seen during training.

[0093] However, the SML method has advantages over the RL method in the following scenarios: For the development of such a solution, a large corpus of image parameter settings historically used (incorporating the operator's knowledge - how to best select illumination) will be available along with all other relevant parameters of the machine, object geometry, and metrology task of interest. Based on this historical database scenario, the previously described simulation engine can run using these optimal illumination parameters (labeled: "optimal because actually used in the past") as well as many other non-optimal illumination parameters (labeled: "sub-optimal because not used in the past") in order to generate labeled training data spanning a wide range of possible illumination settings for the development and testing of the SML model.

[0094] Both the RL and SML methods can optionally be modified to be semi-automatic methods with operator manual intervention. In this case, the operator seeds the process with an educated best-guess illumination setting to start the optimization process. Then, the operator applies boundary conditions to the illumination parameters, for example (knowing that for a given type of object and task, it is advantageous to only use a large-angle annular light sector and not use the central top light nor the back light, so turning them off). For example, this includes removing them from the optimization run (RL) or setting them to zero after the calculation (SML). Then, the operator ranks the candidate illumination settings proposed by the system to manually select and use the best illumination setting and confirm that the derived illumination setting is viable. The operator manually selects a suitable device in cases where the AI model does not converge within an acceptable number of iterations below the KPI threshold (RL) or does not have an acceptable confidence (SML). This serves as a fallback, for example, for certain combinations of object (geometry, material, surface), ambient lighting conditions, and metrology tasks that were not covered in training. Optionally, this can also generate labeled data via user feedback, which can be used to improve the AI / ML model during a retraining run.

[0095] Optionally, the reinforcement learning (RL) agent can be retrained when used in the final product. For example, if the model performs poorly on a particular feature, the model can be retrained on that feature either fully or semi-automatically so that for later lamp adaptation, the RL agent finds the optimal light setting faster.

[0096] Alternatively, instead of a "one-fits-all" model, the implementation can involve machine learning models (RL or SML) specific to certain characteristics of the object or measurement system or certain measurement tasks. For example, regarding specific characteristics of the measurement object, for bright, shiny surfaces, different types of models and / or KPIs can be used than for dark, non-reflective surfaces. Regarding the metrology task of interest, the system can refer to different models when measuring holes, edges, or corners. Regarding the characteristics of the system, the model can be selected based on the model version (i.e., the generation of the metrology device used). Additionally, the model can be selected based on the characteristics of the camera (e.g., grayscale image versus RGB image). For example, the model selection can include selecting a model that assumes a very low level of ambient stray light versus a model that can handle or is insensitive to stray light, or selecting a model that converges very quickly towards an acceptable lighting setup (which may not be optimal) versus a model that converges towards a very likely global best lighting (which may take longer to find). In some of these cases, a machine learning-based classifier can be decisive in determining which model to use. For example, based on the camera's image and / or information about the measurement object available in the metrology SW, a simple binary classifier (e.g., implemented as a decision tree, support vector machine, or neural network) can distinguish between cases of "using the model for bright objects" and "using the model for dark objects".

[0097] Figure 9 An example of an optical measurement system according to the present invention is shown. The system includes at least one training unit and one or more optical measurement devices. Each optical measurement device includes a computing unit, at least one camera, and a plurality of light sources that allow selection of a plurality of different light settings. The computing unit is configured to at least partially control the optical inspection function of the optical measurement system for optical inspection of features of interest, for example, according to the method described above. The training unit is configured to simulate the irradiation of a workpiece model to obtain a synthetic image and, based thereon, train an AI agent or an ML model to automatically control the light sources of the device, for example, according to the method described above. Then, the trained AI agent or ML model is provided to the device.

[0098] The training unit can be a remote system. For each specific device / hardware device, and optionally also for specific conditions at the location of the specific device (e.g., bright object versus dark object, stray light versus no stray light), it can use simulation (and optionally fine-tune with real data) to train a global AI agent / ML model. The AI agent / ML model can be updated and versioned and, for example, be available as a cloud service. Alternatively, the training unit can be provided locally, for example, as part of the optical measurement device.

[0099] Optical measurement devices are local systems that apply global AI agents / ML models according to their specific settings. Optionally, they can record feedback from users (e.g., human evaluation of AI agent results or tagged data including 3D object information and optimally human-defined light settings) for specialized retraining and / or customization. The retraining itself can occur locally (thus benefiting the users of a specific device) or remotely (and potentially integrated into the global AI model to benefit a larger community).

[0100] Although the foregoing has described the invention with reference to some preferred embodiments, it must be understood that many modifications and combinations of the different features of the embodiments can be made. All such modifications are within the scope of the appended claims.

Claims

1. A method (100) for optically inspecting a feature of interest (50) on a workpiece (5), the method using an optical inspection and / or measurement system comprising at least one camera (10) and a plurality of light sources (11), the light sources allowing a plurality of different light settings (45) to be selected, the method comprising: inferring (140) a property of the geometric feature of interest (50) on the workpiece (5) in at least one image captured by the at least one camera (10); controlling (150) the plurality of light sources (11) to select a light setting (45) for illuminating the feature of interest (50) based at least on the inferred characteristic; and while illuminating the feature of interest (50) using the selected light setting (45), optically inspecting (160) the illuminated feature of interest (50) using the at least one camera (10), Features simulating (120) irradiation of a workpiece model (5'); and training (130) an artificial intelligence model to automatically control (150) the plurality of light sources (11) based at least in part on the simulated illumination, The irradiation of the workpiece model (5') is simulated (120) including: Providing (122) 3D data of the workpiece model (5'); accessing (124) a representation (50') of a feature on the workpiece model (5') using at least one virtual camera and a plurality of virtual light sources (11'), wherein the virtual light sources (11') are arranged to correspond to the arrangement of the light sources (11) of the optical measurement system and allow selection of a plurality of light settings (45) identical to the light sources (11) of the optical measurement system; rendering (126) a plurality of basis images (40, 40', 40") imaging a representation (50') of the feature from the viewpoint of the virtual camera under a specific set of predefined light settings (45) of the virtual light source (11'); and obtaining (128) a composite image of all possible light settings (45) of the virtual light source (11') by compositing the basic images (40, 40', 40''), Wherein, training (130) the artificial intelligence model is based on the obtained synthetic image, and controlling (150) the plurality of light sources (11) is automatically performed by the trained artificial intelligence model.

2. The method (100) according to claim 1, comprising: Defining (110) an image quality key performance indicator, said image quality key performance indicator measuring the adequacy of an image of a feature to be used for optically inspecting (160) said feature, in particular wherein defining (110) said image quality key performance indicator comprises automatically deriving said image quality key performance indicator from an image.

3. The method (100) according to claim 2, wherein: Training (130) the artificial intelligence model includes a reinforcement learning loop (30), which is used to train a reinforcement learning agent, in particular using one of a deep Q-network, proximal policy optimization, an advantage actor-critic and an actor-critic with an experience replay algorithm, in particular wherein the reinforcement learning agent is fully or semi-automatically retrained when controlling (150) the plurality of light sources (11).

4. The method (100) according to claim 2 or claim 3, wherein: Defining (110) the image quality key performance indicator includes automatically deriving the image quality key performance indicator from the image and including taking into account brightness and / or contrast around the feature of interest (50); Measuring the density and / or deviation of contour points detected using image processing algorithms around a specific area, curve or polygon; and / or Measures the complexity of contours detected using image processing algorithms.

5. The method (100) of claim 1, wherein: Single-shot parameter estimation is applied in a supervised machine learning framework to estimate optimal light settings (45) directly from an input image acquired with predefined light settings (45), in particular wherein the artificial intelligence model is a classifier or a regressor, in particular having a deep neural network architecture (33).

6. The method (100) according to any one of the preceding claims, wherein: Simulating (120) the illumination of the workpiece model (5') comprises simulating the illumination of a plurality of different workpiece models having a plurality of different geometrical features, the different geometrical features differing at least in their shape and in at least one of material, color, roughness and specular reflection.

7. The method (100) according to any one of the preceding claims, comprising moving a probe (1) of the optical measurement system to approach the feature of interest (50), the probe (1) comprising the at least one camera (10) and at least a first subset of the plurality of light sources (11), wherein Controlling (150) the plurality of light sources (11) comprises controlling the light sources of the first subset, in particular wherein approaching the feature of interest (50) with the probe (1) to infer (140) a property of the feature of interest (50), and / or optically inspecting (160) the feature of interest (50); and / or Controlling (150) the plurality of light sources (11) comprises controlling light sources that are part of a second subset, wherein the light sources of the second subset: is not included in the probe (1), can be controlled individually, and It cannot move relative to the workpiece (5).

8. The method (100) according to claim 7, wherein: The feature of interest (50) is approached by the probe (1) for optically inspecting (160) the feature of interest (50), wherein Training (130) the artificial intelligence model includes training to automatically control the movement of the probe (1) relative to the feature of interest (50) to change the distance and angle of the probe (1) relative to the feature of interest (50), and Controlling (150) the plurality of light sources (11) includes automatically controlling movement of the probe (1) relative to the feature of interest (50) via a trained artificial intelligence model.

9. The method according to claim 7 or claim 8, wherein: The first subset of the light sources (11) is arranged around the at least one camera (10) in one or more concentric rings, in particular at least three concentric rings.

10. The method (100) according to any one of the preceding claims, wherein: The plurality of light sources (11) are arranged into at least eight, in particular at least twelve, individually controllable light groups.

11. An optical measurement system for optically inspecting a feature of interest (50) on a workpiece (5), the optical measurement system comprising a training unit and one or more optical measurement devices, each optical measurement device comprising a computing unit, at least one camera (10) and a plurality of light sources (11), the light sources allowing a plurality of different light settings (45) to be selected, the computing unit being configured to at least partially control an optical inspection function of the optical measurement system for optically inspecting the feature of interest (50), in, The training unit is configured as follows: simulating (120) illumination of a workpiece model (5') to obtain (128) a composite image; training (130) an artificial intelligence model to automatically control (150) the plurality of light sources (11) based at least in part on the obtained (128) composite image; and providing the artificial intelligence model to the one or more optical measurement devices, The irradiation of the workpiece model (5') is simulated (120) including: Providing (122) 3D data of the workpiece model (5'); accessing (124) a representation (50') of a feature on the workpiece model (5') using a virtual camera and a plurality of virtual light sources (11'), wherein the virtual light sources (11') are arranged to correspond to the arrangement of the light sources (11) of one or more of the optical measuring devices and allow selection of a plurality of light settings (45) identical to these light sources (11); rendering (126) a plurality of basis images (40, 40', 40") imaging a representation (50') of the feature from the viewpoint of the virtual camera under a specific set of predefined light settings (45) of the virtual light source (11'); and obtaining (128) a composite image of all possible light settings (45) of the virtual light source (11') by compositing the basic images (40, 40', 40''), wherein each of the one or more optical measuring devices is configured to inferring (140) a characteristic of the feature of interest (50) on the workpiece (5) in at least one image captured by the at least one camera (10); automatically controlling (150) the plurality of light sources (11) via the trained artificial intelligence model and based at least on the inferred characteristic to select a light setting (45) for illuminating the feature of interest (50); and While the feature of interest (50) is illuminated using the selected light setting (45), the illuminated feature of interest (50) is optically inspected (160) using the at least one camera (10).

12. The optical measurement system according to claim 11, wherein: At least one of the one or more optical measurement devices comprises a probe (1) that is movable relative to the workpiece (5) to approach the feature of interest (50), the probe (1) comprising the at least one camera (10) and at least a first subset of the plurality of light sources (11), wherein controlling (150) the plurality of light sources (11) comprises controlling the light sources of the first subset, in particular wherein The feature of interest (50) is accessible to the probe (1) to infer (140) properties of the feature of interest (50) and / or to optically inspect (160) the feature of interest (50); and / or One or more of the optical measurement devices comprises a second subset of light sources not comprised by the probe (1), the second subset of light sources being individually controllable and immovable relative to the workpiece (5), wherein controlling (150) the plurality of light sources (11) comprises controlling the light sources of the second subset.

13. The optical measurement system according to claim 12, wherein: The first subset of light sources (11) is arranged on the probe (1) in one or more concentric rings, in particular at least three concentric rings, around the at least one camera (10); and / or The plurality of light sources (11) are arranged into at least eight, in particular at least twelve, individually controllable light groups.

14. The optical measurement system according to any one of claims 11 to 13, configured to perform the method (100) according to any one of claims 1 to 10, in particular wherein: The artificial intelligence model is a reinforcement learning agent, in particular a deep Q-network or a supervised machine learning agent, in particular a deep neural network (33).

15. A computer program product comprising a program code with computer executable instructions for performing the method (100) according to any one of claims 1 to 10, in particular when run in a system (1) according to any one of claims 11 to 14.

Images