Method and system and automatic example-based parameter estimation in machine vision

Abstract

To provide methods and systems that estimate or improve various parameters of an object recognition model.SOLUTION: A method for updating parameters of an object recognition model includes: providing an object recognition model containing a plurality of model points; providing an image containing at least one object instance; obtaining a direction vector in the image; obtaining transformation parameters that align the model with the object instance, transforming the model point coordinates into image pixel coordinates, and collecting a set of direction vectors for transformed pixels; producing a representative direction vector from the set of direction vectors; determining stable model points from the collected direction vectors and the representative direction vector; and updating the object recognition model by replacing the direction vectors of those model points deemed stable with the representative direction vectors and removing those model points not deemed stable.SELECTED DRAWING: None

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to U.S. Patent No. 7,062,093 B2, entitled "System and Method for Object Recognition," filed September 26, 2001, the entire disclosure of which is incorporated herein by reference.

[0002] FIELD OF THE INVENTION The present invention relates generally to machine vision systems, and more particularly to the visual recognition of objects. [Background technology]

[0003] Background of the Invention Object recognition is part of many computer vision applications. It is particularly useful in industrial inspection tasks, where a model of an object must often be found in an image of the object, resulting in a transform (also called a pose) that encodes the object's position in the image. This transform can be used for a variety of tasks, for example, robot control, pick-and-place operations, quality control, or inspection tasks.

[0004] An object model is defined by various parameters, which may include the object's shape or gray value, the image region to search for instances, the transformation tolerance, pyramid-specific parameters, etc. In current machine vision systems, the generated model can often benefit from further optimization and refinement of such parameters to improve runtime, accuracy, and robustness. However, for most methods, this process of manually editing and refining the model can be complex and time-consuming. For example, many systems provide a graphical mechanism for manually editing the model's contours (see MVTec HDevelop User's Guide, 2022, Chapter 7.3 Matching Assistant: 7.3.3.9 The Tab Creation). It would be desirable to provide a method to automate such refinement steps, such as removing artifacts from the generated model or optimizing transformation parameters. This automation reduces potential errors and the time required to set up the object recognition model, while at the same time resulting in an object recognition model with better performance in terms of runtime and detection than a manually tuned model.

[0005] One set of parameters is the shape of the object. The shape is typically generated from a template image of the object or provided as a contour, for example, from a CAD model, and is represented by a list of model points that define the shape or gray values ​​of the object. One possible optimization is to remove model points that have a low probability of matching in a given use case, or to add points that are reproducible but not yet part of the model. This can include, for example, points that are generated by shadows, reflections, or noise when the model of the object is generated from a template image. In some cases, the CAD model also includes contours that are not clearly defined or visible in the actual image of the object.

[0006] Another set of parameters influences the search process itself. To increase the degree of automation and improve the ease of use of recognition systems, some systems provide methods for automatically determining such parameters. One example of such a machine vision system has already implemented automatic parameter determination to simplify the model creation process (Ulrich and Steger, 2006). Another example speeds up the search phase while improving recognition rates (see MVTec HDevelop Users' Guide, 2022, Chapter 7.3 Matching Assistant). Most of these systems use various discretization levels to speed up the detection process by exhaustively searching only the highest possible level. Potential candidates are identified at the highest possible level and tracked down to lower levels (Steger, 2001). In typical machine vision systems, parameters are estimated using only the original image, if available. The estimated parameter values ​​are then fine-tuned using heuristics to optimize the method's performance and runtime. This can result in suboptimal parameter values, which can lead to the loss of good candidates or the tracking of unnecessary candidates. The latter results in unnecessarily long runtimes. Ideally, a method for finding appropriate parameter values ​​for each of the pyramid levels used is highly desirable for improving the speed, robustness, and accuracy of object recognition models, as well as for enabling the setup of such systems by non-expert users.

[0007] Methods for automating the model refinement process and automatically finding appropriate level-specific parameter values ​​are mostly limited because the model is generated using only a single image or CAD model. Extending the generation method to accept more sample images, rather than just a template image or CAD model, allows for improved object recognition in many aspects. Leveraging sample images results in more stable and improved models. Furthermore, automatic parameter estimation of the creation and search parameters can be extended to automatically estimate values ​​for each of the pyramid levels used. Summary of the Invention [Problem to be solved by the invention]

[0008] Summary of the Invention The present invention provides a method for estimating or improving various parameters of an object recognition model to improve runtime, accuracy, and robustness while minimizing the user interaction required to optimize these values. In one aspect of the present invention, a method is defined for generating an object recognition model with improved contours. A further method for estimating level-specific parameters for an object recognition algorithm is defined by the present invention. [Means for solving the problem]

[0009] According to a first aspect, the present disclosure provides a method for robustly updating parameters of an object recognition model, the method comprising: a. providing an object recognition model, the model including a plurality of model points, each model point being augmented with a coordinate and a direction vector; b. providing at least one digital image, each digital image including at least one object instance; c. For each digital image, obtaining a direction vector for each pixel in the image; d. For each object instance in each digital image, obtaining transformation parameters that align the model with said object instance; e. for each object instance, transforming model point coordinates into image pixel coordinates by applying the transformation parameters of said instance; f. For each model point, collecting a set of direction vectors for all pixels transformed for different instances of the model point; g. for each model point, applying a combination function to the set of all collected direction vectors of said model point to generate a new representative direction vector; h. applying a decision function to the collected direction vectors and representative direction vectors to determine which model points should be considered stable; i. updating the object recognition model by replacing the direction vectors of those model points that are deemed stable with the representative direction vector and removing those model points that are not deemed stable; Includes.

[0010] Preferably, the combination function comprises a sum of all directional vectors, an average over all directional vectors, or a robust estimator of all directional vectors.

[0011] Preferably, the decision function includes a threshold value for the length of the representative direction vector. Preferably, the combination function further calculates a variance value of the set of all collected direction vectors for each model point, and the decision function includes a threshold value for said variance value.

[0012] According to a second aspect, the present disclosure provides a method for robustly creating an object recognition model, the method comprising: a. providing a maximum range of target shapes; b. providing at least two digital images, each digital image including at least one object instance; c. obtaining a feature vector for each pixel in each digital image, the feature vector including a direction vector; d. For each object instance in each digital image, obtaining transformation parameters that align the object instance in a common coordinate frame; e. for each pixel within the maximum range of the target shape, collecting a set of feature vectors for all pixels to which the pixel is transformed for different instances; f. for each pixel within the maximum extent of the target shape, applying a combination function to the set of all collected feature vectors of said pixel to generate a new representative feature vector; g. Applying a decision function to the collected feature vectors and the representative feature vector to determine which pixels should be considered stable; h. creating an object recognition model using those pixels deemed stable along with their representative feature vectors; Includes.

[0013] Preferably, the combination function comprises a sum of all directional vectors, an average over all directional vectors, or a robust estimator of all directional vectors.

[0014] Preferably, the decision function includes a threshold value for the length of the representative direction vector. Preferably, the combination function further calculates a variance value of the set of all collected direction vectors for each model point, and the decision function includes a threshold value for said variance value.

[0015] Preferably, the feature vector calculated in step c further comprises the grey values ​​of the corresponding pixels.

[0016] According to a third aspect, the present disclosure provides a method for robustly updating level-specific parameters of an object recognition model, the method comprising: a. providing an object recognition model; b. providing at least one digital image, each digital image including at least one object instance; c. Providing a set of parameters to be optimized; d. For each digital image, i. creating a multi-level representation of the image corresponding to discretization levels ranging from coarse to fine, wherein at least two different discretization levels are used; ii. searching an object recognition model on a multi-level representation of the image from coarse to fine discretization levels to generate a set of detected instances; iii. for each detected instance, identifying whether it corresponds to an object instance; iv. collecting intermediate values ​​of the parameters optimized for all discretization levels for each detected instance corresponding to the object instance; e. for each discretization level and parameter to be optimized, applying a combination function to the collected intermediate values ​​of the parameter on the pyramid level to obtain a robust estimate of the parameter, and setting the robust estimate as a new parameter in the object recognition model; Includes.

[0017] Preferably, the set of parameters to be optimized includes at least one of a minimum score, a minimum contrast, a maximum overlap of two matches, a range of rotation, a range of scale, or a search area.

[0018] Preferably, transformation parameters are provided for each object instance, said transformation parameters being used in step d.ii to identify which detected instances correspond to the object instance.

[0019] Preferably, an approximate location is provided for each object instance, said approximate location being used in step d.iii to identify which detected object instance corresponds to the object instance.

[0020] Preferably, a number of object instances is provided for each digital image, said number of object instances being used in step d.iii to identify which detected instances correspond to the object instances.

[0021] Preferably, further user input is used to identify which detected instances correspond to object instances in step d.iii.

[0022] Preferably, the combination function is a quantile of the collected median values. Preferably, the combination function uses a probabilistic model to estimate robust parameters based on the collected intermediate values.

[0023] According to a fourth aspect, the present disclosure provides a system comprising a processor, the processor configured to perform a method for robustly updating parameters of an object recognition model according to the first aspect.

[0024] According to a fifth aspect, the present disclosure provides a system comprising a processor, the processor configured to perform a method for robustly creating an object recognition model according to the second aspect.

[0025] According to a sixth aspect, the present disclosure provides a system comprising a processor, the processor configured to perform a method for robustly updating level-specific parameters of an object recognition model according to the third aspect.

[0026] List of Figures The invention will now be described with reference to the following figures. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram of the contour refinement process using two images according to the present invention. [Figure 2] FIG. 10 is a diagram of the contour refinement process using uniform and random direction vectors in accordance with the present invention. [Figure 3] FIG. 2 is a diagram of a contour refinement process using an initial model and at least one input image according to the present invention. [Figure 4] FIG. 10 is a diagram of the contour refinement process using an initial model and an input image with additional edges, in accordance with the present invention. [Figure 5] FIG. 2 is a flow diagram illustrating automatic labeling of an input image according to the present invention. [Figure 6] FIG. 10 is a flow diagram illustrating the determination of level eigenvalues ​​in accordance with the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] Detailed Description of the Preferred Embodiments of the Invention The methods and algorithms described are in electronic form and are considered to be computer-implemented.

[0029] Hereinafter, an image or digital image refers to an image available in electronic form as a two-dimensional array of pixels. An image is typically a representation of the real world captured by an electronic imaging device. Alternatively, an image can be synthetically created, for example, by rendering a three-dimensional object or transforming another image. Each pixel has one or more gray values ​​or intensities associated with it. Typically, these gray values ​​represent the pixel's intensity in the electronic imaging device for a particular set of wavelengths. For example, to represent a color image, multiple gray values ​​per pixel can be used. Each pixel has associated two-dimensional image coordinates (x,y) that represent its location within the two-dimensional array of pixels. Such coordinates can be integer-valued, in which case they represent a complete pixel, or real-valued, in which case they represent the exact location of a subpixel within the image. A region within an image is a set of pixels.

[0030] The gradient of the gray values ​​of an image can be used to define a direction vector and contrast value for each pixel. Various methods exist for estimating gradients (Steger et al., [chapter 3.7.3], 2018). Edges in an image are curves where gray values ​​change significantly. Edges are typically extracted from images using methods such as Canny, Deriche, Sobel, or gray value difference filters (Steger et al., [chapter 3.7.2], 2018). Other methods for calculating gradients or edges can be used without departing from the spirit of the present invention. When calculated from an image, edges are typically stored as a connected list of subpixel-precise coordinates augmented with a contrast value indicating how much the gray value changes from one side of the edge to the other. Contours consist of multiple subpixel-precise points, each with a corresponding direction vector. The direction vector is typically a gradient or normalized gradient vector. Contours can be obtained from edges extracted from an image. Alternatively, a CAD model can be discretized to obtain contours. A contour model is a number of contours that define the outer shape and inner edges of the object instance being searched for. The present invention is not limited to edge features and can be easily extended to line features or interest point features by those skilled in the art.

[0031] A coordinate transformation, translation, or pose is a function that transforms image coordinates from one coordinate frame to another. Typically, such transformations include translation, rotation, isotropic or anisotropic scaling, or general affine transformations. More generally, transformations can include various types of deformations that model deformations of objects. Transformations are parameterized using transformation or pose parameters.

[0032] An object recognition model, or simply a model, is a data structure that contains the parameters necessary to find and locate specific object instances in an image. The appearance of the object must be known in advance and is encoded in the object recognition model. The object instance is located by calculating transformation parameters that align the object instance in the image with the object in a reference coordinate frame.

[0033] An object recognition method is an algorithm that uses an object recognition model to find instances of an object in a digital image and returns transformation parameters that align the object with the found instance (Steger et al., [chapter 3.11], 2018). Such methods are typically divided into an offline phase, in which an object recognition model of an object is generated, and an online phase, in which the object recognition model of an object is found in a search image. The method is typically divided into an offline phase, in which an object recognition model is created for a particular object, and an online phase, in which the model is used to find object instances in images. Note that this is a purely conceptual division; some of the steps can be interleaved without departing from the spirit of the present invention.

[0034] In the offline or model generation phase, an object recognition model is created using a set of parameters. One set of parameters describes the shape of the object, which must be provided by the user. The shape is usually provided in the form of a reference image of the object and a region surrounding the object in the reference image, or as a CAD model of the object. In the case of a reference image, the edges of the image are calculated within the provided region. The resulting edges are optionally filtered by thresholding their contrast. In the case of a CAD model, the edges are provided directly by the CAD model and are optionally sampled. In both cases, the model is generated using a set of model points p i =(p i x , p i y ) Tand the associated direction vector d i =(d i x , d i y ) T , i = 1,...n, where n is the number of points in the model. The points typically sample the visible edges and contours of the object, and the corresponding direction vectors are orthogonal to said edges and contours. Each point may have an additional feature vector representing, for example, the gray value of a color.

[0035] In addition to the object shape, a set of further parameters can be set in the offline phase to further guide the model generation and online phase. Among other things, these parameters typically include the type of transformation allowed between model coordinates and image coordinates. This can include the range of allowed rotations, the range of allowed translations, the range of allowed isotropic or anisotropic scaling, as well as more complex deformation models. A further important parameter is the minimum score, which is a threshold for the similarity between the reference object and the detected instance in order for the instance to be returned to the user. Further possible parameters are described in the following sections.

[0036] In the online or search phase, the user provides a search image along with the object recognition model created in the offline phase, and the method uses the model and its parameters to find an instance of the object in the search image. This process is called template matching or matching. It is performed by iterating through a set of possible and allowed transformation parameters and, for each transformation parameter, calculating a similarity score between the model transformed with that transformation parameter and the search image using a similarity measure. Locations where this similarity score exceeds a defined minimum score and is maximal mark a potential match. The minimum score is typically a user-defined parameter. The set of possible and allowed transformation parameters is calculated based on user-defined parameters such as the range of allowed rotations, the range of translations, the range of isotropic or anisotropic scales, or the search region, as well as logical constraints such as staying within the image boundaries.

[0037] The similarity score of the similarity measure is a metric that represents the similarity between the model transformed into image coordinates and the search image under some parameters. Typically, the similarity score is such that a higher score indicates a higher similarity between the transformed model and the image. Preferably, the similarity score is normalized to be between 0 (no similarity) and 1 (high similarity). Different similarity scores or similarity measures have been proposed in the literature (Steger et al., [chapter 3.11], 2018), where similarity is calculated based on the gray values ​​of the model points and the image pixels at the transformed model points using normalized correlation or normalized cross-correlation. First, the mean and deviation of both sets of gray values ​​are normalized separately. Then, the average pixel-wise similarity of the normalized values ​​is calculated. Normalized correlation is invariant to linear gray value changes between the model and the search image. Preferably, the present invention uses cosine similarity as the similarity measure (see Steger, U.S. Patent No. 7,062,093 B2). This measure calculates the average over pixel-wise dot products between the model's normalized direction vector and the normalized image gradient at the corresponding pixel location. This similarity measure is robust to partial occlusions and clutter, and invariant to non-linear contrast changes.

[0038] For further improvement of the object recognition model, it may be necessary to acquire sample search images with labeled object instances on each image. The label indicates the location of the object instance on the sample image. The label can be provided as a freeform surface region where the object instance resides. Alternatively, the label can be provided as a bounding box around the object instance. The bounding box can be an axis-aligned rectangle or a rotated rectangle. Alternatively, the label can be provided as a transformation parameter between model and image coordinates. In general, the label should be such that it is possible to identify the instance given its pose.

[0039] The visible edges of an object can vary for several reasons, such as variations in the manufacturing process, inherent variations in the object class, or image acquisition artifacts. Thus, not all model points may be visible for all object instances, or their position or orientation relative to the object instances may change. A point is said to be stable if it has a high probability of visibility across all object instances and low variation in relative position or orientation. This definition can be extended to additional feature values, such as gray value. Points with correspondingly low probability or high variation are said to be unstable.

[0040] In most cases, stable points are systematic rather than random. For example, an object's contour may be composed of stable points due to precise manufacturing and image acquisition, but a texture printed on the object may contain unstable points due to inherent variations during the printing process. For this reason, it is beneficial for a model to rely solely or mostly on stable points. This allows for faster and more robust matching and allows the calculated score to be more expressive. However, when creating a model using a single reference image or CAD model, it is impossible to determine which points are stable because they are emergent properties across multiple object instances.

[0041] The present invention uses information from multiple input images or object instances to determine stable points to be used as model points. This information is used to generate a new model using the corresponding stable points or to refine the contours of an existing model by excluding unstable points and including stable points. Furthermore, information from multiple input images or object instances can be used to refine the sub-pixel accurate location, orientation, and feature vectors of stable model points. A model so created or refined is called a stable object recognition model.

[0042] To generate a stable object recognition model, multiple input images are required. The input images can be multiple captures of the same object under different conditions, different instances of the same object, or a mixture of both. To find a stable point, the transformation between the different instances must be known. In one embodiment, the input images are pre-aligned. Object instances in the images are pre-aligned if they appear in the same position and orientation. This alignment can be achieved via mechanical fixturing during acquisition. In another embodiment, if pre-aligned images are not available, a processing step is used to detect object instances in each image and derive a transformation from each instance to a reference position. The images are then aligned to each other using the derived transformation. If an image contains multiple instances of an object, multiple aligned images are created by aligning one instance at a time. As a result, multiple aligned images are obtained in which the object instances are always in the same position.

[0043] The present invention uses the aligned images to identify stable points and calculate their positions and orientations using two functions: a combination function that combines points from multiple aligned images, and a decision function that calculates which points should be considered stable.

[0044] For each pixel location in the aligned images, the combining function collects the directional vectors from all aligned images at that pixel location and calculates a representative directional vector based on the collected directional vectors. In a first embodiment, the combining function simply selects the directional vector from the first image or a randomly aligned image as the representative directional vector. In a preferred embodiment, the combining function calculates the sum over the directional vectors. In another preferred embodiment, the combining function calculates the average over all directional vectors. In another embodiment, the combining function calculates a robust estimator over all directional vectors, such as the median or an iteratively improved robust estimator (Koch, [chapter 3.8], 1999). Figure 1 illustrates the combining function. It shows a schematic representation of an object instance 101 consisting of a rectangle and a small circle within the rectangle. It also shows a schematic representation of an instance 102 consisting of a rectangle and a triangle within the rectangle. The small arrows on the outline depict edge points with directional vectors. The outer rectangular portion of the model represents the stable portion of the instance. The small circles and triangles represent the unstable portions or edges in the input image. After aligning the two instances, a set of direction vectors is obtained for each point, as shown in instance 103.

[0045] In a preferred embodiment, the decision function uses a representative directional vector at each pixel location to determine whether the pixel should be used as a stable point. Figure 1 depicts the decision function for instance 104, where only stable model points at the edges of rectangles are retained, and pixels on circles and triangles are discarded. In one embodiment, the decision function is a threshold on the length of the representative directional vector. For a set of random directional vectors with an estimated sum of zero, the length of the sum is an indication of how well the directional vectors match across all aligned images. Furthermore, pixels that are not part of an edge have directional vectors with short lengths. Thus, the length of the sum over all directional vectors indicates the match of the directional vectors. Similar arguments can be made for other embodiments of the combining function. This is shown in Figure 2, where, for illustrative purposes, the representative directional vector is the sum over all collected directional vectors. In one example, we have a normalized directional vector 202 that indicates approximately the same direction 200 as the normalized input directional vector, resulting in a representative directional vector 203 that is approximately the same length as the normalized input directional vector. If the input directional vector indicates a random direction 201, the representative combined directional vector 204 will be much shorter. For both the summation function and the averaging function, the threshold is a decision function suitable for application to each pixel. In another embodiment, the decision function calculates the variance or standard deviation of the collected directional vectors for each pixel and uses a threshold on the variance to determine whether each pixel is stable. In another embodiment of both the combination function and the decision function, cosine similarity is used instead of Euclidean distance to calculate the mean, median, variance, or standard deviation of the set of directional vectors. One further embodiment of the present invention is to divide the combined directional vector by the number of input images to result in a directional vector with an average amplitude that remains normalized to length 1.

[0046] In all embodiments of the combination function and the decision function, the thresholds can be predetermined or dynamic, such that a fixed number of values ​​are selected or a certain percentage of values ​​are selected.

[0047] In a further embodiment, the intermediate image can be created by using the representative direction vector as the vector value for each pixel, in which a decision function can be applied, for example, for each pixel or to various neighboring pixels, to determine whether each pixel is stable or not based on the distribution of the representative direction vector in the pixel's neighborhood.

[0048] The result of the decision function is a number of stable points or pixels that are used in subsequent processing steps. Applying the decision function minimizes or even eliminates the influence of directional vectors that represent artifacts or unstable parts of the object instance. One typical use of the resulting stable pixels is to generate an object recognition model based on them.

[0049] In one exemplary embodiment, the present invention provides a method for improving an existing object recognition model passed as input by excluding unstable model points and refining its contour to include stable model points that were not included in the initial model. This method accepts an initial object recognition model as input, in addition to the other required inputs in the exemplary embodiments described above. The input image in this case does not need to be pre-aligned, as the initial object recognition model provided as input can be used to automatically align the input image. This is depicted in FIG. 3 , where an initial object recognition model is provided as input, shown as a rectangle with its contour model points 301. An input image 302 is also provided, in which a rectangular object is visible, but additional noisy pixels 303 are also visible. Furthermore, part of the object is occluded by another object 304. After combining the direction vector from the model with the direction vector from the input image, a combined direction vector 305 is obtained. Following the same logic as the no-input-model method, representative direction vectors corresponding to unstable model points have a smaller length. Model points corresponding to unstable model points can be identified and excluded from the initial model, as shown at 306.

[0050] One preferred embodiment of the present invention not only filters out unstable model points but also adds new model points that were not initially included in the model. This allows for the modification of initial models that may not contain important model points, thereby making matching more robust. Here, stable points are found and their corresponding model points are identified. If the initial model did not include them, it can be expanded to include the identified stable points. This is depicted in FIG. 4, where the initial object recognition model 401 does not include the interior contour within the rectangle. Because both input images 402 and 403 include the interior contour 404, the collected direction vectors 405 result in extra stable model points that were not present in the initial model. Therefore, the adapted object recognition model 406 is expanded to include model points that correspond to the stable model points on the interior contour.

[0051] A further embodiment of the present invention is to use the input image to refine the direction vectors of existing model points. In some cases, the initial object recognition model already contains a stable model point, but its direction vector is not accurate. Information from the combined direction vector at this model point can be used to refine the orientation and length of the direction vector.

[0052] To speed up the search, template matching typically uses a multilevel approach, also known as a pyramid approach (Steger, 2001; Steger et al., [chapter 3.11.2], 2018; Tanimoto, 1981). For this purpose, a multilevel representation of both the model and the search image is created. This multilevel representation defines different discretization levels, which are smoothed and subsampled versions of the original image. In a preferred embodiment, the multilevel representation is a mean image pyramid. In another preferred embodiment, a Gaussian image pyramid is applied. These discretization levels are defined from coarse to fine, with the finest, i.e., lowest, level defined as the original image. The search algorithm then begins by performing a full search only on the coarsest or highest level to identify potential candidates. The candidates are then tracked through the discretization levels. Tracking is performed by searching for candidates at the next finer level using a limited set of pose parameters close to the candidate's pose parameters, thus effectively refining the pose parameters. For each level, and for each candidate, a decision is made whether the candidate should be pursued further or discarded. Candidates are pursued until the lowest level is reached, and the remaining candidates, i.e., search results or object instances, are returned. In the following description, recursive subdivision, multi-level representation, and image pyramid are used synonymously.

[0053] Several parameters can be used to control the pyramid method, affecting both the execution time and the robustness and quality of the results: the number of pyramid levels, as well as the zoom factor between two pyramid levels; the number of candidates initially found at the highest level, as well as the number of candidates tracked up to each successive level; the minimum score of candidates at different levels; the highest pyramid level at which the full search is performed, and the lowest pyramid level up to which candidates are tracked; the minimum contrast in gradient to be considered an edge; the maximum overlap of two matches before they are de-duplicated, i.e., considered to be the same match; the range of transformations, including the range of rotation, the range of isotropic or anisotropic scale, the search area, and the range of translation.

[0054] Most of these parameters represent a trade-off between runtime and detection performance. Parameter settings that lead to higher detection rates at the expense of runtime are called relaxed parameters. While relaxed parameters can be used to robustly detect instances in images during the offline phase, they typically lead to prohibitive runtimes during the online phase. In practice, manually tuning these parameters for each object class and use case is difficult, time-consuming, and therefore error-prone and expensive. The present invention provides a method for estimating appropriate parameter values ​​for a specific use case. This makes it possible to create an object detection model with optimal runtime and detection performance for a given use case while minimizing the manual effort required for fine-tuning. To achieve this, prior knowledge of expected search images and object instances is required in the form of labeled data. The method expects at least one input image, each containing at least one instance of an object. Furthermore, an object recognition model should be provided as input. Furthermore, the user can limit the parameters to be optimized.

[0055] In some cases, input images need to be labeled to indicate the location of one or more instances on each image. To obtain this labeled data, the method supports two different possibilities. In a typical embodiment, a user of the method provides labeled data as input to the method. The labels can be obtained using well-known image processing methods, such as segmentation, to find regions that serve as instance labels for object instances. In another embodiment, the method performs a search on each input image using a provided object recognition model, using the detected instances as instance labels. To ensure that all potential instances are found, all parameter values ​​are relaxed before the search. This is depicted in FIG. 5. The method receives as input an object recognition model (step 500) as well as input images for which labels are unavailable (step 501). The method searches the model on each input image (step 502) and visualizes the found instances for the user (step 503). The user is then prompted to verify each instance (step 504) with the possibility of marking each instance as an accepted label (step 505) or an incorrect label (step 506). If the user accepts the instance, it is added to the labeled data (step 505). If the instance is rejected (step 506), the user can decide to label the instance by manually providing it (step 507) or can prompt the method to rerun the search (step 508). In the latter case, the parameters of the model are relaxed before rerunning the method starting from the search phase (step 502).

[0056] After labeled data is acquired, the present invention estimates appropriate values ​​for both the highest and lowest pyramid levels for the current use case. The number of pyramid levels estimated and used is a critical decision that directly affects the runtime and robustness of the object recognition method. The shape of the input model plays a key role in this estimation, since the coarsest level is estimated by selecting the highest possible level that maintains acceptable detail in the model's contours, i.e., the minimum number of points for the model's contours at this level. If the contours are very small at that level, it is difficult to distinguish the object from, for example, clutter. In that case, too many possible matching candidates would have to be evaluated and potentially further tracked. If the estimated highest pyramid level were selected to be as low as possible, a full search at the highest pyramid level would be prohibitively slow. In a preferred embodiment, the method tries multiple possible values ​​for the highest pyramid level. For each such value, the results of the object recognition method are evaluated against the labeled data. The highest pyramid level at which all or a user-acceptable number of labeled data is found is then used as the highest pyramid level. In most cases, the lowest pyramid level is estimated to be level 1, i.e., the original image. However, in some cases, the sample input image may contain high levels of noise, low contrast, or other irregularities, resulting in a loss of correct matches while tracking to the lowest pyramid level. In that case, an appropriate value for the lowest pyramid level is selected by finding the lowest possible level at which all labeled data is still correctly tracked and located. In an alternative embodiment, the highest and lowest pyramid levels are provided by the user.

[0057] After obtaining the highest and lowest pyramid levels, an object recognition method is applied to the input image using the relaxation parameters. This is depicted in FIG. 6, with the obtained image (step 600) and label (step 601) at the top. For each input image, a search of the input object recognition model is applied. In the object recognition method, a search is performed at the highest pyramid level to identify candidates (step 602), which are then tracked through the image pyramid (step 603). One preferred embodiment automatically and recursively relaxes the parameter values ​​until all labeled instances in all input images are found. During the search phase, the calculated level-specific parameter values ​​are stored for each labeled instance, referred to as intermediate values ​​(step 604). In this list of intermediate values, only certain values ​​within the relaxation parameter range appear. One example is that the relaxation parameter for instance rotation allows for a full circle (0 to 360 degrees), meaning that all rotations are searched, but only instances within the range of -5 to +5 degrees are actually found during the search. At the end of the search phase for all input images, an extensive list of observed values ​​for each parameter for each pyramid level is available. The method then determines level-specific parameter values ​​using an appropriate combination function for each parameter (step 605). This process is typically part of an offline process. During online processing of an object recognition system, once a search image is acquired (step 606), the search at the highest pyramid level (step 607) and candidate tracking (step 608) benefit from these level-specific parameter values ​​to return a pose for each found instance (step 609). In the example of a rotation range, the online search phase can constrain the search to accept -5 to +5 degrees instead of a full circle.

[0058] Various parameters, including those related to the generation of a model and those related to the search phase of a model, can benefit from the estimation of level-specific parameter values. Among the parameters related to the generation of a model are the contrast of edge detection, isotropic scaling parameters (isotropic minimum scale, isotropic maximum scale), and anisotropic scaling parameters (minimum row scale, maximum row scale, minimum column scale, and maximum column scale). Regarding parameters related to the search phase, the minimum score for an accepted instance, the maximum accepted overlap between two different instances, and angle parameters (angle start, angle end) can benefit from level-specific estimation. One example of how level-specific values ​​can be beneficial in an object recognition method is the estimation of the minimum score for an instance. When searching a model from coarse to fine, a similarity measure is used to compare an instance with the model at each discretization level to obtain a score. Depending on this score value, the instance is tracked to a finer discretization level. Typically, the score is compared to a minimum score that defines the lowest acceptable score for the instance. Having a level-specific estimate of the score helps optimize this decision of when an instance is worth pursuing further, without risking discarding good candidates or accepting more candidates than necessary.

[0059] Depending on the parameters to be estimated, an appropriate combination function is used to determine the estimate. For some parameters, such as the minimum score of the match, the minimum scale in the row or column direction, or the start angle of the expected rotation, the appropriate value is selected to be the minimum of the parameter values ​​stored at each level. For other parameters, such as the end angle of the expected rotation or the maximum scale in the row or column direction, the appropriate value is selected to be the maximum of the parameter values ​​stored at each level. In further embodiments, the minimum or maximum value can be used after relaxing it by adding (subtracting) or multiplying (dividing) a safety margin. This safety margin can be a probability value based on a modeled distribution of the collected data. In another embodiment, the modeled distribution of the data can be used for some parameters to define a quantile confidence interval. In that sense, the estimate is determined by defining the proportion of labeled instances that must be correctly found in the search phase when this value is used. In further embodiments, the mean or median of the different stored instance values ​​can be used. The mean or median can also be relaxed by adding or subtracting a standard deviation confidence value to ensure that all potential instances are found. In typical embodiments, methods from the field of robust error statistics are used to obtain the appropriate value. Here, outliers are modeled and removed from the distribution. Famous estimators are, for example, the Tukey, Huber, and Gauss methods (Steger et al., [chapter 3.8.1.2], 2018). Furthermore, statistical models include classification methods as well as machine learning methods. In the case of machine learning statistical models, models previously trained on labeled data from previous use cases can be used to increase the robustness of estimated parameter values.

[0060] Other aspects, features, and advantages will become apparent from the above summary and the following description, including the drawings and claims.

[0061] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description is to be considered illustrative or exemplary and not restrictive. It will be understood that changes and modifications may be made by those skilled in the art within the scope of the following claims. In particular, the present invention encompasses further embodiments having any combination of features from the different embodiments described above and below.

[0062] Moreover, in the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single unit may fulfill the functions of several features recited in a claim. Terms such as "essentially," "about," and "approximately" in connection with an attribute or value also specifically define the respective attribute or value precisely. Any reference signs in the claims should not be construed as limiting the scope. REFERENCES C. Steger, M. Ulrich, and C. Wiedemann, Machine Vision Algorithms and Applications, 2 nd ed. Weinheim: Wiley-VCH, 2018. K.-R. Koch, Parameter Estimation and Hypothesis Testing in Linear Models. Berlin Heidelberg: Springer, 1999. Steven L. Tanimoto Template matching in pyramids. Computer Graphics and Image Processing, 16:356-369, 1981. M. Ulrich, C. Steger, US Patent No. 7,953,290, 2006. C. Steger, US Patent No. 7,062,093, 2001. MVTec HDevelop Users’ Guide, Version 22.11, https: / / www.mvtec.com, 2022.

Claims

1. 1. A method for robustly updating parameters of an object recognition model, comprising: a. providing an object recognition model, the model including a plurality of model points, each model point being augmented by a coordinate and a direction vector; b. providing at least one digital image, each digital image including at least one object instance; c. for each digital image, obtaining a direction vector for each pixel in said image; d. For each object instance in each digital image, obtaining transformation parameters that align the model with the object instance; e. for each object instance, transforming the coordinates of the model points into image pixel coordinates by applying the transformation parameters of the instance; f. for each model point, collecting the set of direction vectors for all pixels transformed for the instances in which the model point differs; g) for each model point, applying a combination function to the set of all collected direction vectors of the model point to generate a new representative direction vector; h) applying a decision function to the collected direction vectors and representative direction vectors to determine which model points should be considered stable; i) updating the object recognition model by replacing the direction vectors of those model points that were deemed stable with the representative direction vector and removing those model points that were not deemed stable; A method comprising:

2. The method of claim 1 , wherein the combining function comprises a sum of all directional vectors, an average over all directional vectors, or a robust estimator of all directional vectors.

3. The method of claim 1 or 2, wherein the decision function includes a threshold value for the length of the representative direction vector.

4. 4. The method according to claim 1, wherein the combination function further calculates a variance value of the set of all collected direction vectors for each model point, and wherein the decision function comprises a threshold value on the variance value.

5. 1. A method for robustly creating an object recognition model, comprising: a. Providing a maximum range of target shapes; b. providing at least two digital images, each digital image including at least one object instance; c. obtaining a feature vector for each pixel in each digital image, the feature vector including a direction vector; d. For each object instance in each digital image, obtaining transformation parameters that align the object instance within a common coordinate frame; e. for each pixel within the maximum range of the target shape, collecting the set of feature vectors for all pixels to which the pixel is transformed for the different instances; f) for each pixel within the maximum extent of the target shape, applying a combination function to the set of all collected feature vectors for that pixel to generate a new representative feature vector; g) applying a decision function to the collected feature vectors and the representative feature vector to determine which pixels should be considered stable; h. Creating an object recognition model using those pixels deemed stable along with their representative feature vectors; A method comprising:

6. The method of claim 1 , wherein the combining function comprises a sum of all directional vectors, an average over all directional vectors, or a robust estimator of all directional vectors.

7. The method of claim 1 or 6, wherein the decision function includes a threshold on the length of the representative direction vector.

8. The method of claim 1 , 6 or 7, wherein the combination function further calculates a variance value of the set of all collected direction vectors for each model point, and the decision function includes a threshold value on the variance value.

9. The method according to any one of claims 1 to 8, wherein the feature vectors calculated in step c further comprise the grey values ​​of the corresponding pixels.

10. 1. A method for robustly updating level-specific parameters of an object recognition model, comprising: a. providing an object recognition model; b. providing at least one digital image, each digital image including at least one object instance; c. Providing a set of parameters to be optimized; d. For each digital image: i. creating a multi-level representation of the image corresponding to discretization levels ranging from coarse to fine, wherein at least two different discretization levels are used; ii. searching the object recognition model on the multi-level representation of the image from a coarse to a fine discretization level to generate a set of detected instances; iii. for each detected instance, identifying whether it corresponds to an object instance; iv) collecting intermediate values ​​of said parameters optimized for all discretization levels for each detected instance corresponding to an object instance; e. for each discretization level and parameter to be optimized, applying a combination function to the collected intermediate values ​​of the parameter on the pyramid level to obtain a robust estimate of the parameter, and setting the robust estimate as a new parameter in the object recognition model; A method comprising:

11. The method of claim 1 , wherein the set of parameters to be optimized includes at least one of a minimum score, a minimum contrast, a maximum overlap of two matches, a range of rotation, a range of scale, or a search area.

12. The method of claim 1 or 11, wherein transformation parameters are provided for each object instance, and the transformation parameters are used in step iii to identify which detected instances correspond to the object instance.

13. 13. The method of claim 1, 11, or 12, wherein an approximate location is provided for each object instance, and the approximate location is used in step iii to identify which detected object instance corresponds to the object instance.

14. 14. The method of any one of claims 1 to 13, wherein a number of object instances is provided for each digital image, and wherein said number of object instances is used in step iii to identify which detected instances correspond to object instances.

15. The method of any one of claims 1 to 14, wherein further user input is used to identify which detected instances correspond to object instances in step iii.

16. The method of any one of claims 1 to 15, wherein the combination function is a quantile of the collected mean values ​​or a probabilistic model is used to estimate robust parameters based on the collected mean values.

17. A system comprising a processor, said processor being configured to execute a method for robustly updating parameters of an object recognition model according to the method of any one of claims 1 to 4.

18. A system comprising a processor, said processor being configured to execute a method for robustly creating an object recognition model according to any one of claims 5 to 9.

19. A system comprising a processor, the processor being configured to execute a method for robustly updating level-specific parameters of an object recognition model according to the method of any one of claims 10 to 16.

Images