Method and system in machine vision and parameter estimation based on automatic example

Abstract

The invention discloses a method and system in machine vision and parameter estimation based on automatic examples. The present disclosure provides a method of estimating or improving various parameters of an object recognition model to improve runtime, accuracy and robustness while minimizing the user interaction required to optimize these values. In one aspect of the invention, a method of generating an object recognition model with a refined profile is defined. Additional methods for estimating level-specific parameters for an object recognition algorithm are also defined.

Description

[0001] Cross Reference to Related Applications

[0002] This application is related to U.S. Patent No. 7,062,093 B2, entitled “System and Method for Object Recognition,” filed September 26, 2001, the entire disclosure of which is hereby incorporated by reference herein. TECHNICAL FIELD

[0003] The present invention relates generally to machine vision systems, and more particularly to visual recognition of objects. BACKGROUND

[0004] Object recognition is part of many computer vision applications. It is particularly useful for industrial inspection tasks, in which a model of an object must typically be found in an image of the object, resulting in a transformation (also called a pose) that encodes the position of the object in the image. Such a transformation can be used for various tasks, e.g., robot control, pick-and-place operations, quality control, or inspection tasks.

[0005] A model of an object is defined by various parameters, which can include object shape or grayscale values, image regions to search for instances, allowed range of transformations, pyramid-specific parameters, and others. In current machine vision systems, the generated model can often benefit from further optimization and refinement of such parameters to improve run-time, accuracy, and robustness. However, for most methods, this process of manually editing and refining the model can be complex and time-consuming. For example, many systems provide graphical mechanisms for manually editing the contours of a model (see MVTec HDevelop Users’ Guide, 2022, Chapter 7.3 Matching Assistant: 7.3.3.9 The Tab Creation). It would be desirable to provide a method that automates such refinement steps, e.g., to remove artifacts from the generated model or to optimize transformation parameters. This automation reduces the possible errors and time needed to set up an object recognition model, while at the same time resulting in an object recognition model that has better performance at run-time and detection than a manually tuned model.

[0006] One set of parameters is the shape of the object. This shape is usually either generated from a template image of the object or provided as a contour from, for example, a CAD model, and is represented by a list of model points that define the shape or gray value of the object. One possible optimization is to remove model points that have a low probability of matching in a given use case or to add points that are repeatable but not yet part of the model. This can include, for example, points created by shadows, reflections, or noise in the case where the model of the object is generated from a template image. In some cases, CAD models also contain contours that are not clearly defined or visible in a real image of the object.

[0007] Another set of parameters influences the search process itself. To increase the degree of automation and improve the ease of use of the recognition system, some systems provide methods for automatic determination of these parameters. An example of such a machine vision system has implemented automatic parameter determination to simplify the step of model creation (Ulrich and Steger, 2006). Another example accelerates the search phase while improving the recognition rate (see MVTec HDevelop Users’ Guide, 2022, Chapter 7.3 Matching Assistant). Most of these systems use various levels of discretization to speed up the detection process by only exhaustively searching the highest possible level. Possible candidates are identified on the highest possible level and tracked to lower levels (Steger, 2001). On a typical machine vision system, the parameters are estimated using only the original image, if available. Then, a heuristic is used to fine-tune the values of the estimated parameters to optimize the performance and runtime of the method. This can result in suboptimal parameter values that can lead to the loss of good candidates or the tracking of unwanted candidates. The latter leads to unnecessarily longer execution times. A method that ideally finds the right parameter values for each of the used pyramid levels is highly desirable to improve the speed, robustness, and accuracy of the object recognition model and to allow such a system to be set up by non-expert users.

[0008] A method that automates the model refinement step and automatically finds the right level-specific parameter values is mostly limited because the model only uses a single image or CAD model for generation. Extending the generation method to not only accept a template image or CAD model but also accept more sample images allows for improvements in many aspects of object recognition. With sample images, a more stable and refined model is created. In addition, the automatic parameter estimation of the creation and search parameters can be extended to automatically estimate the values for each of the used pyramid levels. SUMMARY

[0009] The invention provides methods of estimating or refining individual parameters of an object recognition model to improve run-time, accuracy and robustness while minimizing the user interaction required to optimize these values. In one aspect of the invention, a method of generating an object recognition model with refined contours is defined. The invention defines further methods for estimating level-specific parameters of an object recognition algorithm.

[0010] According to a first aspect, the disclosure provides a method for robustly updating parameters of an object recognition model, comprising the steps of:

[0011] a. providing an object recognition model, the model comprising a plurality of model points, each model point augmented with a coordinate and a direction vector;

[0012] b. providing at least one digital image, each digital image comprising at least one object instance;

[0013] c. for each digital image, obtaining a direction vector for each pixel in the image;

[0014] d. for each object instance in each digital image, obtaining transformation parameters aligning the model with the object instance;

[0015] e. for each object instance, transforming the model point coordinates into image pixel coordinates by applying the transformation parameters of the instance;

[0016] f. for each model point, collecting a set of direction vectors of all pixels the model point is transformed onto for different instances;

[0017] g. for each model point, applying a combination function on all collected sets of direction vectors of the model point to produce a new representative direction vector.

[0018] h. applying a decision function on the collected direction vectors and the representative direction vectors to decide which model points should be considered stable;

[0019] i. updating the object recognition model by replacing the direction vectors of those model points considered stable with the representative direction vectors and removing those model points not considered stable.

[0020] Preferably, the combination function comprises an addition of all direction vectors, an average overall direction vector, or a robust estimator of all direction vectors.

[0021] Preferably, the decision function comprises a threshold on the length of the representative direction vector.

[0022] Preferably, the combination function additionally computes a variance of the set of all collected direction vectors for each model point, and wherein the decision function comprises a threshold on the variance.

[0023] According to a second aspect, the disclosure provides a method for robustly creating an object recognition model, comprising the steps of:

[0024] a. providing a maximum extent of a target shape;

[0025] b. providing at least two digital images, each containing at least one instance of an object;

[0026] c. obtaining, for each pixel in each digital image, a feature vector containing a direction vector;

[0027] d. obtaining, for each instance of an object in each digital image, transformation parameters that align said instance of an object in a common coordinate system;

[0028] e. collecting, for each pixel within the maximum extent of the target shape, a set of feature vectors of all pixels that the pixel is transformed onto for different instances;

[0029] f. applying, for each pixel within the maximum extent of the target shape, a combination function to the set of all collected feature vectors of said pixel to produce a new representative feature vector;

[0030] g. applying a decision function to the collected feature vectors and the representative feature vectors to decide which pixels should be considered stable;

[0031] h. using those pixels that are considered stable together with their representative feature vectors to create an object recognition model.

[0032] Preferably, the combination function comprises an addition of all direction vectors, an average overall direction vector, or a robust estimator of all direction vectors.

[0033] Preferably, the decision function comprises a threshold on the length of the representative direction vector.

[0034] Preferably, the combination function additionally computes a variance of the set of all collected direction vectors for each model point, and wherein the decision function comprises a threshold on the variance.

[0035] Preferably, the feature vector computed in step c. additionally contains a grey value of the corresponding pixel.

[0036] According to a third aspect, the disclosure provides a method for robustly updating level-specific parameters of an object recognition model, comprising the steps of:

[0037] a. providing an object recognition model;

[0038] b. providing at least one digital image, each digital image containing at least one object instance;

[0039] c. providing a set of parameters to be optimized;

[0040] d. for each digital image,

[0041] i. creating a multi-level representation of said image corresponding to discrete levels from coarse to fine, wherein at least two different discrete levels are used;

[0042] ii. searching said object recognition model on said multi-level representation of said image from coarse discrete levels to fine discrete

[0043] levels, yielding a set of detected instances;

[0044] iii. for each detected instance, identifying whether it corresponds to an object instance;

[0045] iv. for each detected instance corresponding to an object instance, collecting the intermediate values of the parameters to be optimized for all discrete

[0046] levels;

[0047] e. for each discrete level and each parameter to be optimized, applying a combination function to the collected intermediate values of said parameter on said pyramid levels to obtain a robust estimate of said parameter, and setting said robust estimate as a new parameter in said object recognition model.

[0048] Preferably, said set of parameters to be optimized contains at least one of the following: minimum score, minimum contrast, maximum overlap of two matches, rotation range, scaling range, or search area.

[0049] Preferably, a transformation parameter is provided for each object instance, and in step d.ii, said transformation parameter is used to identify which detected instances correspond to object instances.

[0050] Preferably, an approximate position is provided for each object instance, and in step d.iii, said approximate position is used to identify which detected instances correspond to object instances.

[0051] Preferably, a plurality of object instances is provided for each digital image, and in step d.iii, said plurality of object instances is used to identify which detected instances correspond to object instances.

[0052] Preferably, in step d.iii, additional user input is used to identify which detected instances correspond to object instances.

[0053] Preferably, the combination function is a quantile of the collected intermediate values.

[0054] Preferably, wherein the combination function estimates the robust parameter based on the collected intermediate values using a probabilistic model.

[0055] According to a fourth aspect, the present invention provides a system comprising a processor, wherein the processor is configured to perform the method for robustly updating parameters of an object recognition model according to the first aspect.

[0056] According to a fifth aspect, the present invention provides a system comprising a processor, wherein the processor is configured to perform the method for robustly creating an object recognition model according to the second aspect.

[0057] According to a sixth aspect, the present invention provides a system comprising a processor, wherein the processor is configured to perform the method for robustly updating level-specific parameters of an object recognition model according to the third aspect. BRIEF DESCRIPTION OF DRAWINGS

[0058] The present invention is described with reference to the following drawings, wherein:

[0059] Figure 1 is an illustration of a contour refinement process with two images according to the present invention;

[0060] Figure 2 is an illustration of a contour refinement process with consistent and random direction vectors according to the present invention;

[0061] Figure 3 is an illustration of a contour refinement process with an initial model and at least one input image according to the present invention;

[0062] Figure 4 is an illustration of a contour refinement process with an initial model and an input image with additional edges according to the present invention;

[0063] Figure 5 is a flowchart showing the automatic labeling of an input image according to the present invention; and

[0064] Figure 6 is a flowchart showing the determination of level-specific values according to the present invention. DETAILED DESCRIPTION

[0065] The described methods and algorithms can be considered to be implemented in electronic form and computer-implemented.

[0066] In the following, an image or digital image refers to an image available in electronic form as a two-dimensional array of pixels. An image is typically a representation of the real world acquired by an electronic imaging device. Alternatively, an image can be synthetically created by, for example, rendering a three-dimensional object or transforming another image. Each pixel has one or more gray scale values or intensities associated with it. Typically, those gray scale values represent the intensity of a pixel in the electronic imaging device for a certain set of wavelengths. Multiple gray scale values per pixel can be used to represent, for example, a color image. Each pixel has an associated two-dimensional image coordinate (x, y) representing its position in the two-dimensional array of pixels. Such coordinates can be integer values (in which case they represent complete pixels) or real values (in which case they represent sub-pixel precise positions in the image). A region in an image is a set of pixels.

[0067] The gradient of the gray scale values of an image can be used to define a direction vector and a contrast value for each pixel. There are different methods for estimating the gradient (Steger et al., [Chapter 3.7.3], 2018). An edge in an image is a curve where the gray scale values change significantly. Edges are typically extracted from an image using methods such as Canny, Deriche, Sobel or gray scale difference filters (Steger et al., [Chapter 3.7.2], 2018). Other methods can be used to compute the gradient or edges without departing from the spirit of the invention. If computed from an image, edges are typically stored as a connected list of sub-pixel precise coordinates, augmented with a contrast value indicating how much the gray scale value changes from one side of the edge to the other. A contour is composed of multiple sub-pixel precise points, each with a corresponding direction vector. Direction vectors are typically the gradient or normalized gradient vectors. Contours can be obtained from edges extracted from an image. Alternatively, a CAD model can be discretized to obtain contours. A contour model is a plurality of contours defining the contour lines and inner edges of an instance of an object to be searched. The invention is not limited to edge features, but can be easily extended to line features or interest point features by a person of ordinary skill in the art.

[0068] A coordinate transformation, transform or pose is a function that transforms image coordinates from one coordinate system to another coordinate system. Typically, such a transformation includes a translation, a rotation, an isotropic or anisotropic scaling, or a general affine transformation. More generally, a transformation can contain different types of deformations modeling deformations of an object. A transformation is parameterized using transformation parameters or pose parameters.

[0069] An object recognition model or simply a model is a data structure containing the parameters needed to find and localize instances of a specific object in an image. The appearance of the object has to be known beforehand and is encoded in the object recognition model. An instance of the object is localized by computing the transformation parameters that align the object instance in the image with the object in the reference coordinate system.

[0070] An object recognition method is an algorithm that uses an object recognition model to find instances of an object in a digital image and returns the transformation parameters that align the object with the found instances (Steger et al., [Chapter 3.11], 2018). This method is usually split into an offline phase, in which the object recognition model for an object is generated, and an online phase, in which the object recognition model is used to find instances of the object in search images. Note that this is a purely conceptual split. Some of their steps can be interleaved without departing from the spirit of the invention.

[0071] In the offline phase or model generation phase, an object recognition model is created using a set of parameters. The set of parameters describes the shape of the object that has to be provided by the user. The shape is usually provided in the form of a reference image of the object and a region enclosing the object in the reference image, or as a CAD model of the object. In the case of a reference image, the edges of the image are computed inside the provided region. Optionally, the resulting edges are filtered by thresholding their contrast. In the case of a CAD model, the edges are provided directly by the CAD model and optionally sampled. In either case, the model is represented as a contour of a number of model points and associated direction vectors i = 1,..., n, where n is the number of points in the model. These points usually sample the visible edges and contours of the object, while the corresponding direction vectors are orthogonal to the edges and contours. Each point can have an additional feature vector representing, for example, the gray value of the color.

[0072] In addition to the shape of the object, a set of additional parameters can be set in the offline phase to further manipulate the model generation and the online phase. These parameters usually contain, among others, the type of transformation allowed between the model coordinates and the image coordinates. This can include the range of allowed rotations, the range of allowed translations, the range of allowed isotropic or anisotropic scaling, and more complex deformation models. An additional important parameter is the minimum score, which is a threshold of similarity between the reference object and the detected instance in order to return an instance to the user. Other possible parameters are described in the following sections.

[0073] In the online or search phase, the user provides a search image together with the object recognition model created in the offline phase, and the method uses the model and its parameters to find instances of the object in the search image. This process is called template matching or matching. Template matching or matching is performed by iterating over a set of possible and allowed transformation parameters, and for each transformation parameter, using a similarity measure to compute a similarity score between the model transformed with said transformation parameter and the search image. The position where the similarity score exceeds a defined minimum score and is a local maximum marks a potential match. The minimum score is usually a user-defined parameter. The set of possible and allowed transformation parameters is computed based on user-defined parameters such as the range of allowed rotations, the range of translations, the range of isotropic or anisotropic scales, or the search region, and logical constraints such as staying within the image boundaries.

[0074] The similarity score of the similarity measure is a measure that represents the similarity between the model transformed into image coordinates under some parameters and the search image. Typically, the similarity score is such that a higher score indicates more similarity between the transformed model and the image. Preferably, the similarity score is normalized between 0 (no similarity) and 1 (high similarity). Different similarity scores or similarity measures have been proposed in the literature (Steger et al., [Chapter 3.11], 2018). The similarity is computed based on the gray value of the model point and the gray value of the image pixel at the transformed model point by normalizing the correlation or the normalized cross-correlation. First, the mean and the deviation of both sets of gray values are normalized independently. Then, the mean pixel-wise similarity of the normalized values is computed. The normalized correlation is invariant with respect to linear gray value variations between the model and the search image. Preferably, the present invention uses the cosine similarity as similarity measure (see Steger, US Patent No. 7,062,093 B2). For this measure, the mean over the pixel-wise dot product between the normalized direction vector of the model and the normalized image gradient at the corresponding pixel position is computed. This similarity measure is robust against partial occlusions and clutter and invariant against non-linear contrast variations.

[0075] For further improvement of the object recognition model, it is sometimes necessary to obtain sample search images with the object instances labeled on each image. The label indicates the position of the object instance on the sample image. The label can be provided as a free-form region where the object instance is located. Alternatively, the label can be provided as a bounding box around the instance of the object. The bounding box can be an axis-parallel rectangle or a rotated rectangle. Alternatively, the label can be provided as a transformation parameter between the model and the image coordinates. In general, the label should be such that it allows to identify the instance given the pose.

[0076] The visible edges of an object can vary for a variety of reasons, such as variations in the production process, inherent variations in the object class, or artifacts of the image acquisition. As a result, not all model points are visible for all object instances, or their position or orientation relative to the object instance can vary. We call a point stable if it has a high probability of being visible and a small variation in relative position or orientation across all object instances. This definition can be extended to additional feature values, such as gray value. Points with a correspondingly low probability or large variation are called unstable.

[0077] In most cases, stable points are not random, but systematic. For example, the contour lines of an object can consist of stable points due to precise manufacturing and image acquisition, but the texture printed on the object can contain unstable points to allow for inherent variations during the printing process. For this reason, it is beneficial to have a model that relies on stable points only or mostly. This allows for faster and more robust matching and the computed scores are more expressive. However, when creating a model using a single reference image or CAD model, it is difficult or even impossible to decide which points are stable, as this is an emergent property over multiple object instances.

[0078] The present invention uses information from multiple input images or object instances to determine stable points that should be used as model points. This information is used to generate a new model using the corresponding stable points, or to refine the contour of an existing model by excluding unstable points and including stable points. Additionally, the information from multiple input images or object instances can be used to refine the sub-pixel accurate position, orientation, and feature vector of stable model points. A model created or refined in this way is called a stable object recognition model.

[0079] To generate a stable object recognition model, multiple input images are required. The input images can be multiple acquisitions of the same object, but in different conditions, different instances of the same object, or a mix of both. To find stable points, the transformation between the different instances must be known. In one embodiment, the input images are pre-aligned. An image is pre-aligned if the object instance in the image appears in the same position and orientation. This alignment can be achieved via mechanical fixtures during acquisition. In another embodiment, if pre-aligned images are not available, a processing step is used to detect the instance of the object in each image and to derive the transformation from each instance to a reference position. Then, the images are aligned on each other using the derived transformations. If the images contain multiple instances of the object, multiple aligned images are created by aligning one instance at a time. As a result, multiple aligned images are obtained in the case that the object instances are always in the same position.

[0080] The present invention uses the aligned images to identify stable points and to compute their position and orientation. Two functions are used for this, a combination function that combines the points of multiple aligned images and a decision function that computes which points should be considered stable.

[0081] The combination function collects for each pixel position of the aligned images all the orientation vectors at that pixel position and computes a representative orientation vector based on those collected orientation vectors. In a first embodiment the combination function simply selects the orientation vector of the first or a random aligned image as the representative orientation vector. In a preferred embodiment the combination function computes the sum of the orientation vectors. In another preferred embodiment the combination function computes the average overall orientation vector. In another embodiment the combination function computes a robust estimator overall orientation vector, e.g. the median or the iteratively refined robust estimator (Koch, [Chapter 3.8], 1999). Figure 1 The combination function is depicted. Here one object instance 101 is schematically depicted, which consists of a rectangle and a small circle inside the rectangle. Further, one instance 102 is depicted, which consists of a triangle inside the rectangle. The small arrows at the contour depict the edge points with an orientation vector. The outer rectangular part of the model represents the stable part of the instance. The small circle and the triangle schematically represent unstable parts or edges in the input image. After alignment of the two instances a set of orientation vectors is obtained for each point, as shown for instance 103.

[0082] In a preferred embodiment the decision function uses the representative orientation vector of each pixel position to decide whether this pixel should be used as a stable point. Figure 1 The decision function is depicted in instance 104, where only the multiple stable model points at the edges of the rectangle are kept and the circle and triangle pixels are discarded. In one embodiment the decision function is a threshold on the length of the representative orientation vector. Since for a set of random orientation vectors the sum has an estimate of zero, the length of the sum is an indicator of how much the orientation vectors agree on all aligned images. Further, pixels that are not part of an edge will have orientation vectors with small length. Thus, the length of the sum over all orientation vectors is an indicator of the agreement of the orientation vectors. Similar arguments can be made for other embodiments of the combination function. This is discussed in more detail in Figure 2which, for illustrative purposes, are the sum of all collected direction vectors. In one case, we have normalized direction vectors 202, which are shown in approximately the same direction 200, which results in a representative direction vector 203, which has almost the same length as the normalized input direction vectors. In the case where the input direction vectors are shown in random directions 201, the representative combined direction vector 204 is much shorter. For both the addition and mean functions, the threshold is a suitable decision function applied to each pixel. In another embodiment, the decision function calculates the variance or standard deviation of the collected direction vectors for each pixel, and uses a threshold on the variance to decide whether each pixel is stable. In another embodiment of both the combination function and the decision function, the cosine similarity is used instead of the Euclidean distance to calculate the mean, median, variance or standard deviation of a set of direction vectors. A further embodiment of the invention is to divide the combined direction vector by the number of input images to produce a direction vector with an average magnitude that remains normalized to length 1.

[0083] In all embodiments of the combination function and the decision function, the threshold can be predetermined, or dynamic. A dynamic threshold is such that a fixed number of values are selected, or a certain percentage of values are selected.

[0084] In another embodiment, an intermediate image can be produced by using the representative direction vector as a pixel-wise vector value. In embodiments herein, the decision function can be applied, for example, pixel-wise or on individual neighboring pixels, to decide whether each pixel is stable based on the distribution of representative direction vectors in the pixel neighborhood.

[0085] The result of the decision function is a number of stable points or pixels that will be used in later processing steps. Applying the decision function results in minimizing the influence of, or even excluding, direction vectors that represent artifacts or unstable parts of the object instance. A typical application of the resulting stable pixels is to generate an object recognition model based on them.

[0086] In a typical embodiment, the invention provides a method of improving an existing object recognition model by refining the contour of the existing object recognition model to exclude unstable model points and include stable model points not included in the initial model, passing the existing object recognition model as input. In addition to the other required inputs in the typical embodiment described above, the method receives an initial object recognition model as input. In this case, the input image does not have to be pre-aligned, since the initial object recognition model provided as input can be used to automatically align the input image. This is particularly useful in the case where the input image is a video sequence, and the initial object recognition model is a model of a previous frame in the video sequence. Figure 3In the middle depiction, where the initial object recognition model is provided as input, shown as a rectangle with its contour model points 301. Also provided is the input image 302, where the rectangular object is shown, but additional noisy pixels 303 can also be seen. In addition, a part of the object is occluded by another object 304. After combining the direction vectors from the model with the direction vectors from the input image, we obtain the combined direction vectors 305. Following the same logic as the method without input model, representative direction vectors corresponding to unstable model points will have smaller length. Model points corresponding to unstable model points can be identified and excluded from the initial model, as shown in 306.

[0087] One preferred embodiment of the invention not only excludes unstable model points, but also adds new model points that were not initially included in the model. This allows to correct the initial model that can not contain important model points, which in turn can make the matching more robust. Here, stable points are found and their corresponding model points are identified. If the initial model does not include them, it can be extended to include the identified stable points. This is depicted in Figure 4 In the middle depiction, where the initial object recognition model 401 does not include the inner contour inside the rectangle. Since both input images 402 and 403 include the inner contour 404, the collected direction vectors 405 result in additional stable model points that are not present in the initial model. Thus, the adapted object recognition model 406 is extended to include model points corresponding to the stable model points on the inner contour.

[0088] Another embodiment of the invention is to use the input images to improve the direction vectors of already existing model points. In some cases, a stable model point is already included in the initial object recognition model, but its direction vector is inaccurate. With the information from the combined direction vector at this model point, the orientation and length of the direction vector can be improved.

[0089] To speed up the search, template matching usually employs a multi-level approach, also called pyramid approach (Steger, 2001 ; Steger et al. [Chapter 3.11.2], 2018; Tanimoto, 1981 ). To this end, a multi-level representation of the model and the search image is created. This multi-level representation defines different discretization levels being smoothed versions and sub-sampled versions of the original image. In a preferred embodiment, the multi-level representation is an average image pyramid. In another preferred embodiment, a Gaussian image pyramid is applied. These discretization levels are defined from coarse to fine, where the finest, i.e. lowest, level is defined as the original image. The search algorithm is then initiated by performing a full search on the coarsest or highest level only and identifying potential candidates. The candidates are then tracked through the discretization levels. Tracking is performed by searching for the candidates on the next finer level using a restricted set of pose parameters close to the pose parameters of the candidates, thereby effectively refining the pose parameters. At each level and for each candidate, a decision is made whether this candidate should be tracked further or discarded. The candidates are tracked until the lowest level is reached and the remaining candidates, i.e. the search results or object instances, are returned. In the following description, recursive subdivision, multi-level representation and image pyramid are used synonymously.

[0090] The pyramid approach can be controlled using several parameters, thereby influencing the execution time and the robustness and quality of the results: the number of pyramid levels and the scaling factor between two pyramid levels; the number of candidates initially found on the highest level and the number of candidates tracked to each successive level; the minimum score of a candidate on different levels; the highest pyramid level on which a full search is performed and the lowest pyramid level until which a candidate is tracked; the minimum contrast of gradients that will be considered as edges; the maximum overlap of two matches before they are de-duplicated, i.e. considered as the same match; the range of transformations including the range of rotation, the range of isotropic or anisotropic scale, the search region, and the range of translation.

[0091] Most of these parameters represent a trade-off between runtime and detection performance. Parameter settings that lead to higher detection rates at the cost of runtime are called relaxed parameters. Relaxed parameters can be used to robustly detect instances in images during the offline phase, but usually lead to too high runtimes in the online phase. In practice, it is difficult to manually adjust these parameters for each object class and use case, is time consuming and therefore prone to errors and expensive. The present invention provides a method to estimate appropriate parameter values for a specific use case. This allows to create an object detection model that has optimal runtime and detection performance on a given use case, while requiring minimal human work for fine-tuning. To achieve this, prior knowledge about the expected search images and object instances in the form of labeled data is required. The method expects at least one input image, each image containing at least one instance of an object. Additionally, an object recognition model will be provided as input. Further, the user can restrict the parameters that should be optimized.

[0092] In some cases, the input images need to be labeled to indicate the location of the instance or instances on each image. To obtain this labeled data, the method supports two different possibilities. In a typical embodiment, the user of the method provides the labeled data as input to the method. The labeling can be obtained using well-known image processing methods such as segmentation to find regions that act as instance labels for instances of an object. In another embodiment, the method uses the provided object recognition model to perform a search in each of the input images using the detected instances as instance labels. To ensure that all potential instances are found, all parameter values are relaxed before the search. This is depicted in Figure 5 The method accepts an object recognition model as input (step 500) as well as input images for which no labels are available (step 501). The method searches the model on each input image (step 502) and visualizes the found instances for the user (step 503). The user is then prompted to validate each instance using the possibility to label each instance as an accepted label (step 505) or an incorrect label (step 506) (step 504). If the user accepts the instance, the instance is added to the labeled data (step 505). If the instance is rejected (step 506), the user can decide to label the instance by manually providing the instance (step 507) or prompt the method to rerun the search (step 508). In the latter case, the parameters of the model are relaxed (step 502) before rerunning the method starting from the search phase.

[0093] After obtaining the labeled data, the present application estimates a suitable value for the highest pyramid level and the lowest pyramid level in the current use case. Here, the number of pyramid levels that will be estimated and used is a key decision that directly influences the run-time of the object recognition method as well as its robustness. The shape of the input model plays a major role in this estimation, as the coarsest level is estimated by choosing the highest possible level at which the contour of the model retains acceptable details (i.e., the smallest number of points of the contour of the model at this level). If the contour is very small at that level, it is difficult to distinguish the object from e.g. clutter. In that case, too many possible matching candidates must be evaluated and potentially tracked further. If the estimated highest pyramid level is chosen lower than its possible highest pyramid level, the full search at the highest pyramid level is too slow. In a preferred embodiment, the method tries multiple possible values for the highest pyramid level. For each such value, the results of the object recognition method are evaluated against the labeled data. The highest pyramid level at which all labeled data or a user-acceptable number of labeled data is detected is then used as the highest pyramid level. In most cases, the lowest pyramid level is estimated to be level 1, i.e. the original image. However, in some cases, the sample input image can contain a high level of noise, low contrast or other irregularities that lead to loss of correct matches when tracking to the lowest pyramid level. In such cases, the suitable value for the lowest pyramid level is chosen by finding the lowest possible level at which all labeled data is still correctly tracked and found. In an alternative embodiment, the highest and lowest pyramid levels are provided by the user.

[0094] After obtaining the highest pyramid level and the lowest pyramid level, the object recognition method is applied to the input image using a relaxation parameter. This is depicted in Figure 6 , in Figure 6The top of Figure 6 shows the acquired images (step 600) and the labels (step 601). On each input image, a search is applied of the input object recognition model. In the object recognition method, the search is performed at the highest pyramid level to identify candidates (step 602), which are subsequently tracked through the image pyramid (step 603). One preferred embodiment automatically and recursively relaxes the parameter values until all instances of the labels in all input images are found. During the search phase, the computed level-specific parameter values (step 604), called intermediate values, are stored for each instance of a label. In this list of intermediate values, only certain values in the relaxation parameter range will occur. One example can be the relaxation parameter for the rotation of an instance that allows a full circle (0 to 360 degrees), meaning that all rotations are searched, but during the search, only instances in the range of -5 degrees to +5 degrees are actually found. At the end of the search phase for all input images, a broad list of observed values for each parameter at each pyramid level is available. The method then employs a suitable combination function for each parameter to determine the level-specific parameter value (step 605). So far, this process is typically part of an offline step. During the online processing of the object recognition system, when a search image is acquired (step 606), the search (step 607) and tracking of candidates (step 608) at the highest pyramid level benefit from these level-specific parameter values to return the pose of each found instance (step 609). In the example of the rotation range, the search can be limited to accept -5 degrees to +5 degrees instead of a full circle in the online search phase.

[0095] Different parameters can benefit from the estimation of level-specific parameter values, including parameters related to the generation of the model and parameters related to the search phase of the model. Among the parameters related to the generation of the model are the contrast of the edge detection, the isotropic scaling parameters (isotropic minimum scale, isotropic maximum scale), and the anisotropic scaling parameters (minimum row scale, maximum row scale, minimum column scale, and maximum column scale). For the parameters related to the search phase, the minimum score to accept an instance, the maximum overlap between two different instances, and the angular parameters (angle start, angle end) can benefit from level-specific estimated values. An example of how level-specific values can be beneficial in the object recognition method is the estimation of the minimum score of an instance. When searching the model from coarse to fine, instances are compared to the model using a similarity measure at each discretization level, which yields a score. Depending on this score value, the instance is further tracked down to a finer discretization level. Typically, the score is compared to a minimum score that defines the lowest acceptable score of an instance. Having a level-specific estimate of the score helps to optimize this decision without the risk of discarding good candidates or accepting more candidates than needed when the instance is worth further tracking.

[0096] Depending on the parameter to be estimated, a suitable combination function is used to determine the estimate. For some parameters, e.g. the smallest fraction of matches, the smallest scale in row or column direction, or the start angle of the expected rotation, a suitable value is chosen as the minimum of the parameter values stored at each level. For other parameters, e.g. the end angle of the expected rotation or the maximum scale in row or column direction, a suitable value is chosen as the maximum of the parameter values stored at each level. In further embodiments, the minimum or maximum can be used after relaxing the minimum or maximum by adding (subtracting) or multiplying (dividing) by a safety margin. This safety margin can be a probability value based on a modeled distribution of the collected data. In another embodiment, the modeled distribution of the data can be used for some parameters to define a quantile confidence interval. In this sense, the estimate is determined by defining a percentage of the marked instances that must be correctly found in the search phase when using the value. In further embodiments, the mean or median of the stored instance values can be used. The mean or median can also be relaxed by adding or subtracting a standard deviation confidence value to ensure that all potential instances are found. In a typical embodiment, methods from the field of robust error statistics are used to obtain a suitable value. Here, outliers are modeled and removed from the distribution. Famous estimators are for example the methods of Tukey, Huber, and Gauss (Steger et al., [Chapter 3.8.1.2], 2018). In addition, statistical models include classification methods as well as machine learning methods. In the case of statistical models after machine learning, models trained previously on marked data from previous use cases can be used to increase the robustness of the estimated parameter values.

[0097] Other aspects, features, and advantages will become apparent from the above summary and from the following description (including the drawings and claims).

[0098] While the application has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. It is understood that changes and modifications can be made by those of ordinary skill within the scope of the following claims. In particular, it is within the scope of the application to cover any embodiments with any combination of features from different embodiments described above and below.

[0099] Furthermore, in the claims the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single unit can fulfill the functions of several features recited in the claims. The terms "essentially", "about", "approximately" and the like in connection with an attribute or a value particularly define exactly the attribute or exactly the value, respectively. Any reference signs in the claims should not be construed as limiting the scope.

[0100] References

[0101] C. Steger, M. Ulrich, and C. Wiedemann, Machine Vision Algorithms and Applications, 2nd Edition, Weinheim: Wiley-VCH, 2018. K.-R. Koch, Parameter Estimation and Hypothesis Testing in Linear Models. Berlin Heidelberg: Springer, 1999. Steven L. Tanimoto Template matching in pyramids. Computer Graphics and Image Processing, 16:356-369, 1981.

[0102] M. Ulrich, C. Steger, US Patent No. 7,953,290, 2006.

[0103] C. Steger, US Patent No. 7,062,093, 2001.

[0104] MVTec HDevelop Users’Guide, Version 22.11, https: / / www.mvtec.com, 2022.

Claims

1. A method for robustly updating the parameters of an object recognition model, comprising the steps of: a. providing an object recognition model, said model comprising a plurality of model points, each model point augmented with a coordinate and a direction vector; b. providing at least one digital image, each digital image comprising at least one object instance; c. for each digital image, obtaining a direction vector for each pixel in said image; d. for each object instance in each digital image, obtaining transformation parameters that align said model with said object instance; e. for each object instance, transforming said model point coordinates into image pixel coordinates by applying the transformation parameters of said instance; f. for each model point, collecting a set of direction vectors of all pixels that said model point is transformed onto for different instances; g. for each model point, applying a combination function over all collected sets of direction vectors of said model point to produce a new representative direction vector. h. applying a decision function to the collected direction vectors and the representative direction vectors to decide which model points should be considered stable; i. updating said object recognition model by replacing the direction vectors of those model points considered stable with said representative direction vectors and removing those model points not considered stable.

2. The method of claim 1, wherein said combination function comprises an addition of all direction vectors, an average overall direction vector, or a robust estimator of all direction vectors.

3. The method of claim 1 or 2, wherein said decision function comprises a threshold on the length of said representative direction vector.

4. The method of claim 1, 2 or 3, wherein said combination function additionally computes a variance of all collected sets of direction vectors for each model point, and wherein said decision function comprises a threshold on said variance.

5. A method for robustly creating an object recognition model, comprising the steps of: a. providing a maximum extent of a target shape; b. providing at least two digital images, each digital image comprising at least one object instance; c. for each pixel in each digital image, obtaining a feature vector comprising a direction vector; d. for each object instance in each digital image, obtaining transformation parameters that align said object instance in a common coordinate system; e. for each pixel within the maximum extent of said target shape, collecting a set of feature vectors of all pixels that said pixel is transformed onto for different instances; f. for each pixel within the maximum extent of said target shape, applying a combination function to all collected sets of feature vectors of said pixel to produce a new representative feature vector; g. applying a decision function to the collected feature vectors and the representative feature vectors to decide which pixels should be considered stable; h. using those pixels considered stable together with their representative feature vectors to create an object recognition model.

6. The method of claim 1, wherein said combination function comprises an addition of all direction vectors, an average overall direction vector, or a robust estimator of all direction vectors.

7. The method of claim 1 or 6, wherein the decision function comprises a threshold on the length of the representative direction vector.

8. The method of claim 1, 6 or 7, wherein the combination function additionally computes a variance of the set of all collected direction vectors for each model point, and wherein the decision function comprises a threshold on the variance.

9. The method of any one of claims 1 to 8, wherein the feature vector computed in step c. additionally contains the gray value of the corresponding pixel.

10. A method for robustly updating level-specific parameters of an object recognition model, comprising the following steps: a. providing an object recognition model; b. providing at least one digital image, each digital image containing at least one object instance; c. providing a set of parameters to be optimized; d. for each digital image, i. creating a multi-level representation of the image corresponding to discrete levels from coarse to fine, wherein at least two different levels of discretization are used; ii. searching the object recognition model on the multi-level representation of the image from the coarse to the fine level of discretization, resulting in a set of detected instances; iii. identifying for each detected instance whether it corresponds to an object instance; iv. for each detected instance corresponding to an object instance, collecting the intermediate values of the parameters to be optimized for all levels of discretization; e. for each level of discretization and each parameter to be optimized, applying a combination function to the collected intermediate values of the parameter over the pyramid levels to obtain a robust estimate of the parameter, and setting the robust estimate as the new parameter in the object recognition model.

11. The method of claim 1, wherein the set of parameters to be optimized contains at least one of: a minimum score, a minimum contrast, a maximum overlap of two matches, a range of rotations, a range of scaling, or a search region.

12. The method of claim 1 or 11, wherein for each object instance a transformation parameter is provided, and in step iii. the transformation parameter is used to identify which detected instances correspond to object instances.

13. The method of claim 1, 11 or 12, wherein for each object instance an approximate position is provided, and in step iii. the approximate position is used to identify which detected instances correspond to object instances.

14. The method of any one of claims 1 to 13, wherein for each digital image a plurality of object instances is provided, and in step iii. the plurality of object instances is used to identify which detected instances correspond to object instances.

15. The method of any one of claims 1 to 14, wherein in step iii. an additional user input is used to identify which detected instances correspond to object instances.

16. The method of any one of claims 1 to 15, wherein the combination function is a quantile of the collected intermediate values, or a robust parameter is estimated based on the collected intermediate values using a probabilistic model.

17. A system comprising a processor, wherein the processor is configured to perform a method for robustly updating parameters of an object recognition model according to any one of claims 1 to 4.

18. A system comprising a processor, wherein the processor is configured to perform a method for robustly creating an object recognition model according to any one of claims 5 to 9.

19. A system comprising a processor, wherein the processor is configured to perform a method for robustly updating level-specific parameters of an object recognition model according to any one of claims 10 to 16.

Images