Method and apparatus for determining whether a given image section of a first image appears in a second image.

The method addresses the alignment challenges of existing fingerprint methods by using translation-invariant image feature comparison and a voting weight matrix, enabling reliable identification of image sections on objects with uncertain positions and irregular shapes.

DE102023205147A1Pending Publication Date: 2024-11-28ROBERT BOSCH GMBH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
DE102023205147
Authority / Receiving Office
DE · DE
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2023-06-01
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing image matching methods, particularly fingerprint methods, are not translation invariant, leading to incorrect results when the region of interest (ROI) is not aligned perfectly, and require significant effort to align, making them impractical for objects without defined edges or those with uncertain positions.

Method used

A method that determines unique image features using a calculation rule invariant to translation, allowing for efficient identification of image sections by comparing features across images, even with slight positional variations, and includes a voting weight matrix to determine the presence of an image section with high confidence.

Benefits of technology

The method provides robust and efficient identification of image sections, overcoming alignment challenges and ensuring accurate matching even with small positional offsets, suitable for objects with irregular shapes and uncertain positions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A method for determining whether a given image section of a first image appears in a second image. The method begins by determining (S21) at least one image feature for a plurality of pixels within the given image section. This is followed by determining (S24) at least one image feature for a plurality of pixels in the second image. Subsequently, the determined image features of the second image are compared (S25) with the determined image features of the first image to see if they have identical values. The method is characterized in that a rotationally symmetric surface is determined from the second image, this surface is transformed into a rectangular area, and the image features of the pixels of the second image are calculated from pixels within the transformed area.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for determining whether a predetermined image section of a first image occurs in a second image, as well as to a device, a computer program and a machine-readable storage medium. State of the art

[0002] In production, for example of supplier parts for the automotive industry, complete traceability of the individual parts used is increasingly sought, and in some cases is even required by law or standard.

[0003] This traceability enables faster and more precise conclusions about the causes – for example, in the case of malfunctions. For example, it is possible to identify the manufacturing machine and the process parameters used. It is also possible to identify or isolate other supplier parts affected by the defect, and a recall campaign can be specifically limited to the end products actually affected.

[0004] This traceability in production can be practically ensured by: a. Marking of the respective individual part, today often with a DMC, see https: / / de.wikipedia.org / wiki / DataMatrix-Code, e.g. as laser engraving, needle embossing, printing or sticker. b. Markerless tracking based on the individual surface texture of the component.

[0005] In both cases, a sensor, such as a camera, captures the part to be tracked at a predetermined location on the surface to automatically read the DMC or capture the texture. The texture is then compared with the previously captured texture at approximately the same location and stored (e.g., in a database with a unique index) for all relevant parts. In other words, the aforementioned database comparison compares a current query image with previously acquired reference images.

[0006] The texture comparison yields a high match if the part is the same and a very low match otherwise. For example, the comparison is performed with 100, 10,000, or 1 million parts. Typically, the comparison must be clock-neutral, meaning it must be performed quickly enough to avoid slowing down the production cycle—for example, 10,000 comparisons in a maximum of 200 ms.

[0007] Markerless tracking has several advantages: i. Saving effort, costs, and investments. For example, a laser engraving machine is much more expensive, especially to purchase, than a camera including lighting for texture capture. ii. Laser engraving of DMCs produces smoke, small flakes from the surface that can contaminate the component (e.g., a nozzle) and thus impair its functionality. In contrast, markless tracking is completely clean. iii. Some components are too small or do not have a suitable surface for applying the DMC, while for texture capture even the smallest surfaces 1mm 2 sufficient. iv. While the DMC can be read and used by unauthorized persons with malicious intent, this is not possible with the texture, as only the manufacturer knows the texture's assignment. This makes the process more secure. v. Protection against counterfeiting. While a counterfeit part can easily be provided with a copied DMC, it is physically virtually impossible to clone the component's random surface texture.

[0008] DE 10 2019 210 580 discloses a method for verifying the authenticity of an object. A user photographs an object, such as a banknote, using a smartphone (or app) and shortly thereafter receives confirmation from the app if it is authentic. The texture comparison takes place, for example, in the cloud, where the reference textures are stored in a database and can be used for comparison. Advantages of the invention

[0009] The first and most important advantage of the proposed method is its translation invariance. This means that the identification is invariant (independent) with respect to a translation of a query object in the camera image. In other words: The identification leads to the same result regardless of where the ROI (region of interest) is shifted in the second image – of course, only if the ROI fully contains the correspondence to the first image area. The translation invariance applies to both axes (x, y) of the camera image. The previous requirement in the state of the art to be able to reproduce the position of the reference ROI in the query image is eliminated. The alignment step required by fragile fingerprint methods can therefore be omitted with the method.

[0010] The second advantage is scalability in the sense that the user has the option of restricting the permissible (x, y) translation. In principle, the method is capable of identifying an object even if there is only a small overlap with respect to the object between the reference image and the query image. The user can then restrict the translation area (in the sense of a search area) to the required size, e.g., 20 pixels each to the left, right, top, and bottom. In this example, the translation vector may therefore lie within a 41 × 41 pixel area. The advantage of this restriction is a saving in computational effort (time), since this increases with the translation area area.

[0011] However, it increases much less than is the case with known fingerprint methods.

[0012] These advantages regarding translation allow the application of the proposed method to object types that were previously not manageable with fingerprint methods, e.g., rough castings or parts that lack usable visual references (edges, corners, holes, etc.) that would enable a suitable alignment (preprocessing step for uniform alignment of the query image).

[0013] The third advantage of the proposed method is that its similarity measure (which will be described below) proves to be much more discriminatory in practice than the Hamming distance measure of the fingerprint methods.

[0014] A fourth advantage is that the proposed method can be used with small image dimensions in practice. Even for the most difficult object types, image dimensions of, for example, 400x400 pixels are more than sufficient. In practice, image dimensions of approximately 50x50 pixels are often sufficient for reliable identification. Disclosure of the invention

[0015] In a first aspect, the invention relates to a computer-implemented method for determining whether a predetermined image section of a first image occurs in a second image. The image section is either manually preselected or determined in some other way. The image section is small, particularly relative to the first image; particularly preferably, the image section comprises 50x50 pixels up to 400x400 pixels of the first image.

[0016] The method begins with determining image features for a plurality of pixels within the specified image section. The image features are determined according to a specified calculation rule, whereby the calculation rule calculates the image feature depending on the pixel values ​​of the neighboring pixels. For example, an environment with 3x3 neighboring pixels up to 127x127 pixels around the respective pixel can be used for the image feature. The calculation rule outputs a unique value, and thus the image feature is unique, i.e., the image feature characterizes the observed pixel arrangement of the observed pixel including its surroundings. The calculation rule is selected such that the unique image feature is obtained with a high probability even if the pixel values ​​are at least slightly altered, e.g., due to sensor noise or lighting differences.The calculation rule does not have to determine a unique value for every pixel combination, but uniqueness should be guaranteed with a high degree of probability. Preferably, the calculation rule is at least slightly translation-invariant.

[0017] This is followed by obtaining the second image, in particular a section (ROI) of the second image, which is smaller than the second image but at least as large as the image section of the first image.

[0018] This is followed by determining image features for a plurality of pixels of the second image or within the ROI. The image features of the first and second images are calculated identically, i.e., the same calculation rule is used for the image features.

[0019] This is followed by a comparison of the determined image features of the second image with the determined image features of the first image, preferably taking into account not only the values ​​of the image features but also their relative arrangement to one another in the respective image. If a group of at least two dissimilar image features is found in the second image that are in the same or almost the same arrangement to one another as an identically valued group of dissimilar image features in the first image, it is output that the specified image section is present in the second image; otherwise, it is output that the specified image section is not present in the second image.

[0020] A special feature of the method is that a rotationally symmetrical surface, in particular a ring-shaped or circular surface, is determined from the second image and in particular the first image, and this surface is transformed into a quadrangular, in particular rectangular quadrangular, surface, and that the image features of the pixels of the second image and in particular of the first image within the transformed surface are calculated. The rotationally symmetrical / ring-shaped surface can be transformed into its polar coordinate version with predetermined dimensions (height x width) using a given center point, a 'start' radius, an 'end' radius, and preferably a start / end angle. Accordingly, it should be noted that the predetermined image section originates from the transformed surface of the first image.

[0021] It is conceivable that those neighboring pixels of the rotationally symmetric surface, which are now separated from each other in the quadrilateral surface at opposite ends of the quadrilateral surface, are included in the calculation rule when calculating the image features of these originally neighboring pixels. In other words, it can be provided that the quadrilateral surface is supplemented at its ends with, for example, a copy of the pixels at the opposite ends.

[0022] It is proposed that after the step of determining the image features of the first image, an assignment step takes place which assigns to each of the image features a relative position of the pixel for which the respective first image feature was determined with respect to a reference position of the predetermined image section. The reference position can be a center point of the image section. In the comparison step, if the values ​​of the image features of the first and second images are identical, the assigned relative position of the image feature of the first image is taken and a pixel position of the pixel of the second image for which the identical image feature is present is shifted relative to said relative position and, in particular, stored at this shifted pixel position in a voting weight matrix.This is followed by a pixel-by-pixel aggregation of the shifted, stored positions, whereby if a predominant majority of the output positions with respect to a confidence measure are present at a substantially identical pixel position of the second image, it is output that the specified image section is present in the second image, otherwise it is output that the specified image section is not present in the second image.

[0023] Essentially the same pixel position can be understood as a deviation of + / - 2, 3 or 4 pixels or more.

[0024] Optionally, an output step can follow in which the pixel position with the most counts after aggregation is output as the position of the image section in the second image.

[0025] Optionally, to facilitate the evaluation, the aggregation can be smoothed, which allows a clear maximum to be achieved at a unique location while simultaneously suppressing insignificant secondary maxima.

[0026] Preferably, the image features are used to address a lookup table, with the associated relative position stored for each address. Optionally, the lookup table contains a weighting in addition to the relative positions. The weighting changes, in particular decreases, with increasing distance between the image feature position and the reference position. The weighting can also be used during aggregation to perform a weighted aggregation.

[0027] If multiple image sections are present, each of the process steps described above is performed for each of the different image sections. Preferably, the image sections are evenly distributed across the object texture in the first image, e.g., evenly distributed across a circle.

[0028] It is further proposed that a plurality of the image sections have a predetermined arrangement relative to one another within the first image, wherein the comparison step, in addition to the stored positions, determines how these positions are arranged relative to one another and compares the two arrangements with one another. The arrangement can be given by defined distances between the positions or topological patterns. When the topological patterns are compared, preferably only the structure of the pattern is compared and an orientation of the pattern is not taken into account. The advantage here is that an additional independent decision criterion of a topological arrangement effectively makes the statement as to whether the object depicted in the images is identical more reliable.

[0029] It should be noted that the step of determining the image features of the pixels of the second image occurs immediately after capturing the second image, preferably in a computing unit, and is transmitted to a server, which transmits a signal back depending on the comparison. Preferably, the image features of the image section have already been determined in the cloud or transmitted there and are simply compared with received image features in the cloud. Alternatively to the cloud, the method can be executed locally.

[0030] Preferably, in the first aspect of the invention, the first and second images depict a surface of the same workpiece.

[0031] The first and second images can be taken with the same camera or cameras. There are no special requirements for the cameras used to capture the two images; a low image resolution is sufficient. The recording setup should be as similar as possible in both cases with regard to the choice of lens, distance to the selected area of ​​the object, focus on it, and type and arrangement of lighting. Exposure time and illumination intensity should be selected so that the respective image does not exhibit any significant motion blur and is neither too dark nor overexposed. If it is not possible to use two identical camera setups, the differences can be subsequently compensated for using image processing.For example, if the second camera has a different distance, focal length, or pixel pitch than the first camera, the resulting scaling difference can be compensated for by image scaling so that the viewed area of ​​the object is then approximately the same size (in pixels).

[0032] In further aspects, the invention relates to a use of one of the methods of the first aspect for checking whether a surface image of an object belongs to a given object. Preferably, the query object is compared with all reference objects. The best match (highest overall confidence value) is then determined. If this value is high enough (above a threshold), the match is output. However, if the best overall confidence value is not high enough, no match has been found. The query object is either a forgery or has not yet been recorded as a reference.

[0033] In further aspects, the invention relates to a use of one of the methods of the first aspect for verifying an authenticity of a product.

[0034] In further aspects, the invention relates to a device and a computer program, each of which is configured to carry out the above methods, and to a machine-readable storage medium on which this computer program is stored.

[0035] Embodiments of the invention are explained in more detail below with reference to the accompanying drawings. In the drawings: Fig. 1 schematically shows images of produced components, each with a plurality of image areas AL in a grid-like arrangement; Fig. 2 schematically shows an embodiment of a method for recognizing image areas.

[0036] Several methods already exist for markerless tracking based on surface texture, such as the so-called Track & Trace Fingerprint from the Fraunhofer Institute for Physical Measurement Techniques. This technology is based on the fact that many semi-finished products or components have a microscopically unique surface structure or color texture. A defined area of ​​the component is selected and imaged in high resolution using an industrial camera. From the image recording, including its specific structures and their positions, a numerical identifier is calculated and assigned an ID. This pairing is stored in a database along with other data such as measurement or production dates. For later identification, the entire process is repeated, and a data comparison returns the ID and other individual characteristics of the component.

[0037] The aforementioned "calculated numerical identifier" is referred to below as a fingerprint. The fingerprint can be imagined as a number with many digits or, equivalently, as a long bit sequence.

[0038] During the aforementioned database comparison, the current fingerprint is compared with previously recorded reference fingerprints, and the degree of dissimilarity is determined in each case. Ideally, exactly one reference fingerprint is found that shows little dissimilarity from the query fingerprint, while the comparison with all other reference fingerprints reveals a high degree of dissimilarity. This is precisely when the identification is considered successful.

[0039] The difference between the physical reference object and the physical query object is determined representatively from the pair of fingerprints alone, for example as the Hamming distance of the two bit sequences or according to another distance measure definition.

[0040] At this point, the present invention differs particularly clearly from the prior art.

[0041] The methods for traceability and trademark protection known in the state of the art are based on the creation of a so-called fingerprint, which describes the individual texture properties for an ROI (region of interest) of the object, for example in a bit string of fixed length, which can be several thousand bits.

[0042] The texture comparison is then performed by comparing the fingerprints. For example, the Hamming distance between the fingerprint of the reference object and the fingerprint of the query object is calculated, i.e., a bit-by-bit comparison is made, and the differences are counted. The smaller the Hamming distance (ideally 0), the more similar the objects are.

[0043] If the Hamming distance is normalized with the length of the bit string, a similarity value between 0 and 1 is obtained, whereby in practice the values ​​only use the range 0 to about 0.5, since with statistical independence of the bit strings only about half of the bits differ.

[0044] Comparing fingerprints, e.g. using Hamming distance, has the advantage of speed, so many comparisons can be carried out in a short time.

[0045] However, it also has a significant disadvantage: the result of the fingerprint comparison is not translation-invariant. For example, if the fingerprint for the query object is not created at the same location (ROI) as the reference object, but offset by 20 pixels, for example, it will bear virtually no resemblance to the reference fingerprint.

[0046] In this case, the texture comparison produces an incorrect result: although it is the same object, the texture image pair is misclassified as dissimilar.

[0047] In practice, this unfavorable behavior due to the lack of translation invariance of the fingerprint methods has led to a large amount of additional effort being required to ensure that the position of the ROI in the reference and query images is as identical as possible, because deviations of just a few pixels are no longer tolerable.

[0048] For some object types, the geometry of the part allows the object to be mechanically positioned in a reproducible position (pose) relative to the camera. This creates additional effort (robot gripper arm to grip the object, placing the object in a suitable template in front of the camera, re-gripping and replacing the object, time required, cost, vulnerability). For other object types, these geometric constraints are not met, and visually prominent features of the object type must be exploited to make the ROI reproducible, e.g., the position of holes, grooves, corners, edges, etc. This also creates additional effort (adaptation of image analysis algorithms to the component, application by specialists).

[0049] However, there are also object types for which neither one nor the other method works to reproducibly find an ROI.

[0050] This applies, for example, to objects of a coarser nature, such as metallic castings, which do not have defined edges, but rather have strongly bevelled and rounded edges due to the ability to remove them from the casting mold and, possibly making things even more difficult, random casting burrs.

[0051] Even with flat object types such as sheet metal, paper, cardboard, fiber-reinforced plastic panels, etc., the reproducibility of the position of ROls can be difficult or even impossible, especially if the edges are bent, folded, punched or trimmed during processing, thus losing the possible visual references.

[0052] Minor uncertainties in the location of the ROIs can be compensated for by creating multiple fingerprints with slightly offset ROI positions (e.g., shifted pixel by pixel). However, this multiplies the effort required for fingerprint generation and comparison accordingly, often even quadratically, because the uncertainty exists in both axes (x, y). Larger uncertainties regarding the location of the ROIs can practically not be addressed.

[0053] For the fingerprint methodology to work in operation, both steps must always be successful: the alignment step (also called matching or alignment step) and the fingerprint comparison step.

[0054] What makes this particularly difficult is that the alignment step, which must be performed at each camera station (the first station for initial acquisition and a number of additional stations for identification) for ROI alignment, must always produce the same result. This is difficult to guarantee in practice, especially in continuous operation.

[0055] Accordingly, fingerprinting methods are considered fragile in practice and are therefore unpopular with users. For example, a slight shift of the camera relative to the placement template can result in the components no longer being identifiable at this identification station, thus bringing production to a standstill.

[0056] In the following, the following confidence measures will be introduced, which evaluate the confidence in the quality of a found correspondence.

[0057] Confidence measure q B is mathematically defined as: qB=p1p2−1. Here, p1 is the height of the best peak and p2 is the height of the second-best peak. Peak here refers to the value of the global or local maximum in the voting weight matrix or in a further processed form of the voting weight matrix.

[0058] A preferred further processing of the voting weight matrix is ​​smoothing it after the accumulation (collecting the voting weights) is complete. A suitable smoothing filter is, for example, a two-dimensional Gaussian smoothing filter.

[0059] Since p1 ≥ p2 is always satisfied, q B ≥ 0.

[0060] The confidence measure q B is suitable, for example, to assess whether the correspondence search was successful or not from a single image comparison (1-to-1 comparison: 1 query image or query ROI against 1 reference image area). If the correspondence was found, q B large (e.g. q B= 11), otherwise q B small (e.g. q B = 0.1).

[0061] In an alternative embodiment for the confidence measure, the subtraction of 1 could be omitted.

[0062] An alternative, even simpler confidence measure q A is: q A = c · p1 with an optional constant c. Thus, the height of the best peak is used as a confidence measure.

[0063] This again refers to the value of the global maximum in the voting weight matrix or in a further processed form of the voting weight matrix (e.g., after smoothing filtering). The optional constant c can be used to define the range of values ​​of q. A to set, for example, 0 ≤ q A ≤ 1.

[0064] The confidence measure q AThis has the advantage that the determination of the second peak p2 is omitted, thus saving computational operations. On the other hand, it also eliminates the reference value for normalization.

[0065] However, this apparent disadvantage becomes irrelevant in a 1-to-N comparison (1 query image against many (N) reference images, as will be discussed in more detail later), because there is usually a hit that is characterized by a high q A -value, compared to N - 1 non-hits, all of which have low q A -values. This information is completely sufficient to determine the winner, so the height of the second-best peak in the same query image is dispensable.

[0066] The simpler confidence measure q A will therefore be the preferred confidence measure below.

[0067] For reliable operation of the invention, a camera is preferably used that captures a two-dimensional image in the form of grayscale or multidimensional values ​​(color values) of a (partial) surface of the object (so-called texture capture), and this image should be as reproducible as possible from camera to camera. In other words, the image should appear as similar as possible. Easily correctable differences, which can be corrected, for example, by rotating, scaling, shifting, or adjusting the brightness of the image content, do not pose a problem.

[0068] To ensure that the image content of the two shots of the same subject is as similar as possible, the two shooting setups should be as similar as possible or at least coordinated. This applies particularly to the lighting.

[0069] This is relatively easy to achieve for objects with a matte surface, as the incident light is widely scattered and thus the image of the object hardly changes if the lighting position varies slightly. For objects with a highly reflective surface, e.g. ground or brushed and then chrome-plated steel, the image depends heavily on the position of the light source, especially if the light source is point-shaped. To ensure that both recording setups still produce reproducibly similar images of the object, it is highly advisable to choose lighting in which the light hits the surface as evenly as possible from all directions (diffuse lighting). This can be achieved, for example, with dome lighting close above the object, which illuminates the inside of a matte white hemisphere, creating as undirected indirect lighting as possible.The camera is located inside the hemisphere or looks into it through a hole in the area of ​​the hemisphere's axis of symmetry.

[0070] Especially for such difficult surface types, it can be useful to calculate the texture image from several consecutive images taken by the same camera, each under different lighting conditions. For example, four images can be taken in quick succession, each with only one quadrant of a ring- or dome-shaped diffuse illumination switched on. Suitable, well-known methods include shape from shading or photometric stereo.

[0071] For example, shape-from-shading calculates a so-called texture image that describes the surface's reflectivity, as well as a curvature image that describes the local curvature. Both images are suitable for this invention, either independently or combined, as they each provide a reproducible image of the object in which the lighting dependencies have been eliminated.

[0072] Various methods are known for finding an image area from a first image in the second image, e.g. from DE 10 2019 210 580.

[0073] One of these methods first defines a small outline of an image section, such as a square, circular, or polygonal outline.

[0074] The feature points lie within this outline: these are image features formed from their respective surroundings, each represented, for example, with 16 bits. Such an image feature therefore represents a compact representation of a more or less local image content. Overall, a larger overall image area contributes to the totality of the formed features. A latch descriptor can be used to calculate the image feature for a given pixel: G. Levi and T. Hassner. LATCH: Learned arrangements of three patch codes. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9, 2016. It should be noted, however, that there are numerous other known methods for calculating these features.

[0075] In this example, the square is 50 × 50 pixels in size and contains 2500 feature points. In other words, the feature density here is 1 feature per pixel. However, it could also be higher or lower, e.g., 4 features per pixel or 1 feature per 9 pixels.

[0076] The respective feature value (with, for example, a 16-bit word length) serves as an address in a lookup table, into which the feature position in the image relative to a reference position in the image is entered, optionally also further information, e.g. a weight.

[0077] At the latest when all 2500 feature points have been entered into the table (a smaller number is usually sufficient), the processing of the viewed image area from the first image is completed.

[0078] The second image is the image for which the position corresponding to the reference position is to be determined. The search area for this position can cover the entire second image. The features are then generated accordingly for the entire second image. Alternatively, the search area only covers part of the second image (ROI). Features are then generated for the second image for the area corresponding to the enlarged ROI, whereby this is enlarged by the permissible displacement, e.g. by 20 pixels to the left, right, top and bottom. For the second image, either the exact same method for calculating features is used as for the first image, or an adapted method that appropriately takes into account the change in the state of the texture, e.g. from dry and clean to wetted with oil.

[0079] In this case, prior knowledge can be used which, for example, describes the change in texture in the image during the transition from dry to oil-wetted or vice versa, for example statistically, for example in the form of a composite distribution density of the local contrasts (or color values) between the two states.

[0080] In this example, the feature density in the second image is again 1 feature per pixel. It may be higher or lower and does not have to match the feature density of the first image.

[0081] The features generated from the second image are again used to address the lookup table, but this time in a read-only manner. As described above, starting from the current feature position in the second image, a vote weight is cast at a relative position determined from the lookup table and entered into a vote weight image. This can be done for each feature in the search area, with the votes being collected (accumulated) in the vote weight image, preferably additively or weighted additively.

[0082] In a preferred embodiment, the size of the voting weight image approximately corresponds to the size of the search area. The voting weight image can also be selected to be smaller or larger. However, it need not be selected to be larger than the range of the relative position during voting. This range depends on the size of the first image area and the relative location of the reference position—or in other words, on the limited length of the 2D vector.

[0083] In the methods for retrieving an image region, a correspondence in the voting weight image results in a very distinct and spatially focused cluster point whose position corresponds to the desired position, i.e., the correspondence of the reference position from the first image in the second image. By evaluating the cluster point, e.g., by determining its maximum position or its center of gravity position, the desired correspondence position can be determined with sub-pixel accuracy, if desired or necessary. Depending on the camera resolution, this can correspond to sub-millimeter accuracy.

[0084] The determined position information can be used advantageously in a variety of ways: From the difference between the reference position in the first image and the found position in the second image, it is possible to determine how the pose between camera and object differs between the two camera stations. Statistics can be calculated from this. The mean value provides information about the differences from camera station to camera station that you might want to correct. The deviation from the mean value, e.g. expressed as the standard deviation, provides information about how the poses fluctuate from object to object. These position fluctuations can be caused by variations in gripping with the robot or by differences in the shape of the parts. For example, such variations can be large in undeburred cast parts.

[0085] The statistics can also be helpful for parameterizing search spaces. As described above, it may be advisable not to fully utilize the translational degree of freedom, but rather to configure a smaller search space in favor of lower computational effort. However, it should not be chosen too small to avoid missing any correspondence.

[0086] If several correspondences are searched for per image pair, conclusions can be drawn from the arrangement of the found correspondence positions and the confidence of the classification can be further improved.

[0087] The following section examines how advantages can be gained from the optional use of multiple reference image areas.

[0088] The multiple reference image regions are placed in the reference image. For example, M = 12 reference image regions are placed relative to the reference image so that they preferentially capture those portions of the object's surface that enable differentiation. Noise-like textures resulting from random processes are particularly well suited for recognizing individual objects. In contrast, regions that look virtually identical for every object would be unsuitable. Unsuitable areas for positioning the reference image regions include, for example, overexposed image regions that are so bright that the camera can no longer resolve details, or very dark image regions where the signal-to-noise ratio is low, so that the image signal is largely determined by sensor noise (which, of course, cannot contribute anything to identification).

[0089] Furthermore, unsuitable areas can include, for example, nearly homogeneous image regions where the signal is approximately constant or varies only slightly locally, thus providing little or no information for identification. Other areas are also very blurry image regions that lie significantly outside the camera's focal plane or focal area. Other areas are image regions that look the same for every object and therefore do not serve to distinguish them, such as edges, corners, and holes. This also includes cast shadows caused by lighting.

[0090] Also unsuitable are surfaces that are destroyed during the tracking of the object or that undergo serious changes that make tracking impossible, for example surfaces that are milled, ground or sandblasted, or surfaces to which opaque paint is applied or that are covered in any other way so that they are no longer visible later.

[0091] The definition of reference image areas is usually done once for a component type, after the acquisition setup (camera, lighting, pose, image resolution, image section) has been determined. This definition is usually the only parameterization step required. The definition of reference image areas then normally applies to all objects of this component type. This can be done by an expert or automatically.

[0092] For example, the definition can be made by an expert, using M = 12 reference image areas, whose centers are placed on a circle at equal angular intervals of 30°, like the full hours on a clock face. This arrangement is suitable, for example, for a circular or annular object surface. The reference image areas are preferably located within a well-focused, randomly textured area. Edges and blurred image areas are avoided.

[0093] Overlaps of the reference image areas are permissible, but should be avoided, as there is no additional benefit from the additional effort.

[0094] As an overall confidence measure q Ges When using multiple image areas, in the simplest case the sum or mean of the M confidence measures is calculated, for example: qTotal=1M∑i=1Mqi, where the individual confidence measure q i here preferably according to q A or according to q B or may be formed according to another rule.

[0095] The ability to place several small reference image areas independently on suitable partial areas, each with a suitable surface texture, and thus to achieve an optimized adaptation to the component type in terms of distinguishability with little effort, represents an advantage of this method.

[0096] The use of an expert to determine the reference image areas may be disadvantageous for cost or availability reasons.

[0097] Alternatively, there is the option to automatically determine the optimal image areas. For this purpose, a training approach is provided that requires a very small number of objects, e.g., N R = 10 pieces, but at least N R = 2 pieces.

[0098] In the following it is assumed that of each of these N R Training objects each have a reference image (hence the index R) and N Q ≤ N R including one query recording each (index Q).

[0099] Let 1 ≤ N Q ≤ N R fulfilled, or in other words: Ideally, there is also a query recording for each object, so N Q = N R , but if necessary, a single query recording is sufficient, i.e. N Q = 1.

[0100] For example, the training runs as follows: 1. A predetermined number M of reference image areas are initially placed, e.g. randomly or regularly, e.g. as tiling. 2. This placement will be applied to all N R reference images. Then, the procedure for pairwise comparison of the N Q Query images with the N R Reference images are applied. This creates N R × N Q Similarity values ​​that are stored in a similarity matrix of size N R × N Q where the column index corresponds to the query numbering and the row index corresponds to the reference numbering.

[0101] In the case of N R = N Q and if the order is identical, the correct assignments would be on the main diagonal. 3. The similarity matrix is ​​evaluated. The goal is to optimize the similarity matrix, achieving the maximum value for correct assignment in each column (usually on the main diagonal), while keeping all other values ​​in the column as small as possible. The measure can be, for example, the quotient or the difference between the highest similarity and the second highest similarity (in the respective column), averaged across all columns. This results in an overall score for the placement setup, which must be maximized. 4. If the total score reaches a maximum or exceeds a threshold, the optimization is complete, meaning the determined parameters have been optimized. Otherwise, the optimization continues. 5. The setup is varied. This includes changing the positions of the individual reference image areas (i.e., their location in the image), and / or the dimensions of the individual reference image areas (e.g., for rectangular or elliptical shapes: width, height, aspect ratio, angle), and / or the number of reference image areas.

[0102] It should be noted that overlaps of the reference image areas are again avoided as much as possible.

[0103] Of course, common optimization methods can be used to accelerate this training or to guide it to a global optimum. For example, gradients are calculated to identify and prioritize the parameters that have the greatest influence on the overall score.

[0104] It is advantageous to also include the computational effort in the overall rating score (less computational effort leads to better ratings). This automatically ensures that the number and dimensions of the reference image areas are not larger than necessary, and that they do not overlap.

[0105] Objects are often constructed with essentially rotational symmetry. This often makes it impossible to consistently position the object at the same angle (around the axis of rotational symmetry) in front of the camera. A notch on the side, for example, would provide clarity and thus help, but such a tool is not available for some object types.

[0106] For such and similar objects that are ambiguous in this sense with respect to rotation around an axis, the invention offers a significant advantage over the prior art, because their translation invariance in two directions can be converted into tangential invariance and axial invariance. Tangential invariance, which can also be referred to as rotation invariance, is particularly important here.

[0107] For this purpose, it is suggested that a section of the image, for example, a circular area, be digitally rolled. To do this, the two radii of the circular disk and the position of its center point in the image must be determined. This point should at least approximately coincide with the intersection point of the rotational symmetry axis in the image.

[0108] The parameters for cropping the circular disk are either known in advance or can be determined from the image. In particular, it may be useful to redetermine the center point for each shot if it varies, e.g., based on an estimate of the center of symmetry. In this case, an approximate estimate, e.g., accurate to within a few pixels, is sufficient.

[0109] When the circular disk is digitally unrolled, it is cut open and unrolled into a rectangular shape. The pixel values ​​(gray values, color values) are warped accordingly.

[0110] It should be noted that rolling is not limited to surfaces such as circular arcs; the surface to be rolled can also be an inner / outer cylinder, or a truncated inner or outer cone. Even shapes such as a barrel shape, dumbbell shape, or wave shape are conceivable. Thus, rolling can generally be applied to rotationally symmetrical surfaces.

[0111] At the intersection line, the rectangle can be imagined as continuing cyclically on the right and left, because these two ends belong together. Accordingly, all image processing operations and further processing steps can continue beyond this intersection line without interruption.

[0112] After unrolling the circular disk-shaped image content into a rectangular image, the procedure can be applied as usual: In other words, an unrolled query image is compared with an unrolled reference image.

[0113] If the described principle of cyclical continuation is consistently followed, the result of the identification according to the invention is independent of where the radial cut was made on the circular ring.

[0114] Assuming the round component is in an unknown angular position, the rolling process transforms the unknown angular position into an unknown horizontal translation. However, the algorithm is invariant to this. This means the result is independent of the horizontal translation. The object can therefore be identified in its unrolled form, despite the unknown angular position. Thus, the rolling process transforms the translation-invariant method into a rotation-invariant method.

[0115] The unrolled image also exhibits vertical translation invariance. This means that even vertically shifted correspondences can be found. The vertical translation invariance translates into invariance in the radial direction. This is also extremely useful because it can compensate for an inaccurately placed center point. In other words, if the estimate of the center of symmetry was inaccurate, e.g., misplaced by 5 pixels, and the circular disk is unrolled around the wrong center point, this is not a problem because the (then vertically misplaced) correspondences can still be found. This provides the method with a further enormous gain in robustness.

[0116] In summary, this means in other words that for the identification of essentially rotationally symmetric objects in practice, their angular position may be arbitrary and unknown when the image is taken and that an approximate knowledge or estimate of the position of the rotation axis or the center is already sufficient.

[0117] This important advantage over the state of the art is made possible by the given two-dimensional translation invariance of the method, which can be converted into rotation invariance and invariance in the radial direction by rolling.

[0118] Invariance in the radial direction is not true invariance in the mathematical sense, because the rolling is a nonlinear mapping with a scaling factor that depends on the respective radius. An error in the position of the center point translates accordingly into a scaling error. However, this apparent problem has proven to be negligible and therefore irrelevant in practice.

[0119] As soon as several reference image areas are used per image, an optional additional possibility arises to further increase the confidence in the identification.

[0120] Because the M reference image regions are arranged in a specific order in the reference image, it is expected that the correct correspondences found in the query image are in the same or a similar order. This can be verified to create an additional or alternative quality measure.

[0121] If the arrangement corresponds to the expectation, the confidence is high; deviations from the expectation reduce the confidence accordingly.

[0122] If not all M possible correspondences can be found, e.g., because the object has suffered damage in some places, at least the remaining correspondences should still be in the expected partial arrangement. This should still lead to a relatively high confidence.

[0123] This illustrates Fig. 1. On the reference object 10 in , M = 12 image regions are defined, labeled here with the letters A - L, which are to be used for texture-based identification. The centers of the image regions are located in a predefined arrangement 11, which is also shown graphically to the right of the image. In this case, it is a grid-like arrangement with a rectangular grid with constant grid spacing (in both dimensions). Of the 4 × 4 possible grid positions here, the four corner positions are unused.

[0124] Other arrangements of the image areas are conceivable, e.g. on a circle or other geometric shapes.

[0125] At the time of the query image, the same object 12 already exhibits surface damage, making identification difficult. Other factors, such as changes in imaging conditions, may further complicate the process.

[0126] All this leads to the fact that some correspondences, e.g. G, are not found at all and some correspondences, e.g. F, have a very low confidence or a strongly reduced confidence, e.g. B.

[0127] It is proposed, especially for such difficult cases (which may be significantly more difficult than shown here), the possibility of introducing an additional check in which the consistency of the arrangement is checked in order to derive an additional evaluation metric.

[0128] This will be explained using the present illustrative example. The correspondences E and I found here are considered particularly strong. The positions of these correspondences, for example, can then serve as anchor points to anchor the known arrangement, in this case the grid, accordingly, as shown in arrangement 13 on the right. This leads to a slight rotation of the grid to the right (because the object in the query image is rotated accordingly). A slight scaling (reduction or enlargement) of the grid may also be necessary here.

[0129] Starting from the grid anchored in this way, all other expected correspondence positions can be determined, e.g., that of B. If this correspondence is actually found within a radius around this expected position, this should have a positive effect on the combined confidence, thus increasing it. The radius takes into account, for example, the expected measurement inaccuracies, image distortions, numerical inaccuracies, and error propagation. The radius can have a fixed or variable radius. Optionally, the radius grows with the distance to the anchor points, which can also be seen in arrangement 13, where the circles have different diameters. The circles optionally also grow more the closer the anchor points are to each other (reduced leverage, not shown graphically here).

[0130] The circumcircles are not to be taken literally; they don't necessarily refer to circular areas; they can also have other shapes. Elliptical shapes are particularly relevant, as these result from the calculation of errors.

[0131] The combined confidence can then be calculated as follows: 1. Choose two found correspondences, e.g. E and I, to anchor the grid (here synonymous with expected arrangement) at their positions by appropriately translating, scaling and rotating the grid. 2. Check all other found correspondences {A - D, F - H, J - L} to see if they fit the anchored grid, e.g. if they lie within their respective expected radius. 3. In the simplest case, count the fulfillment of expectations. This could result in a count value from 0 to M - 2 = 10 as a new confidence measure, with the anchor points themselves not being counted.

[0132] In an improved version, the counting would be replaced by a summation of the M individual confidence values, each of which is calculated according to q A or q B are determined. Again, only for those correspondences that lie within the respective radius. Optionally, the individual confidence values ​​of the anchor points can also be added.

[0133] The formula for this could be, for example: qGes=1M∑i=1Mwi⋅qi, where w i = 1, if the i-th correspondence lies in the expected radius or is an anchor point, otherwise w i = 0.

[0134] In a further improved version, the proximity criterion would be defined more flexibly, i.e., not as a yes / no decision (is within the proximity or not), but rather with an evaluation of the distance of the respective found correspondence position from the respective expected position. For example, with a maximum value for a direct hit, a medium value for a slightly larger distance, and approximately 0 for too large a distance.

[0135] The formula for q is also suitable for this Ges , but not with a binary weighting factor, but with a scalar weighting factor w i , which evaluates the position of the i-th correspondence.

[0136] As a value of w iFor example, the sampling of a two-dimensional correlated or uncorrelated Gaussian density function or a two-dimensional conical density function can be used, each with its maximum at the expected position. The evaluation number w obtained by sampling the density function at the location of the found correspondence i can be multiplied by the respective individual confidence value q i (e.g., peak height). These products, up to M - 2 = 10, are then added together again. Optionally, the anchor points can also be included in the total, preferably with the perfect score.

[0137] Optionally, the latter steps can be repeated with all other possible anchor points, for example, next with anchor points C and L. The further the two anchor points are apart, the greater the leverage for anchoring the grid, which is advantageous.

[0138] These permutations avoid the risk that incorrect anchoring due to faulty correspondence leads to an inappropriately poor overall assessment. Of all the total confidences q obtained by permutation, Ges It is therefore preferable to continue using the one with the highest achieved value, because it is most likely that the underlying anchoring was correct.

[0139] For the example considered here (with M = 12 image areas), it is easily possible to consider the entire set of possible permutations. If only a subset of the permutations is to be considered, it is advantageous to favor anchor points with high individual confidence values ​​and long levers.

[0140] So far, only the case with two anchor points has been considered, which already allow the adjustment of rotation and scaling between the reference and query images. However, it is also possible to work with a different number of anchor points, e.g., three or four. This even makes it possible to allow and consider different camera perspectives on the object surface, for example. This gives the user more freedom when installing the system. For example, they no longer have to ensure that the camera's viewing direction at the respective station is perpendicular to the object surface; oblique viewing angles and even handheld camera operation are also possible.

[0141] Fig. 2 shows a flowchart of a method for recognizing image regions.

[0142] The method begins with determining (S21) image features for a plurality of pixels within the specified image section. Optionally, an assignment S22 is also performed. In this case, a relative position of the pixel for which the respective first image feature was determined is assigned to each of the image features with respect to a reference position of the specified image section. This is followed by obtaining (S23) the second image, determining a rotationally symmetrical surface from the second image and transforming this surface into a rectangular surface, and determining (S24) image features for a plurality of pixels of the second image. The transformation was also performed for the first image, as explained above.

[0143] This is followed by a comparison (S25) of the determined image features of the second image with the determined image features of the first image to determine whether both have an identical value. If at least one pair of identical image features is present, it is output that the specified image section is present in the second image; otherwise, it is output that the specified image section is not present in the second image. It should be noted that for image features that are, for example, only conditionally unique, at least two or more pairs of identical image features should preferably be present. Preferably, the majority of the pairs of identical image features are located at the same position in the voting weight matrix.

[0144] If step S24 has been executed, in the comparison step (S25), if the values ​​of the image features of the first and second images are identical, the associated relative position of the image feature of the first image can be taken and a pixel position of the pixel of the second image for which the identical image feature is present can be stored, shifted relative to the said relative position. As explained above, these determined positions can be understood as a voting weight, which is summed pixel by pixel in a voting weight matrix (voting image). Using the voting weight matrix, a position of the image section in the second image can be determined.

[0145] The two most important areas of application for the procedure according to Fig. 2 are: Markerless tracking (traceability) based on the individual surface texture, e.g. of components in industrial production. Brand protection through the use of the individual surface texture of products to recognize originals from one's own production and to differentiate / detect third-party products or counterfeits.

[0146] In both cases, the naturally existing texture of an object's surface is exploited, which is random in nature and physically practically impossible to clone.

[0147] The latter property, namely that random processes create a unique, non-clonable surface texture for each object, can be exploited for trademark protection. To this end, the manufacturer captures the texture for each produced object at a predefined location (e.g., photographed or scanned) and stores it, e.g., in a database in the cloud.

[0148] To later verify the authenticity of an object, its texture is scanned again at approximately the same location, and a database comparison is performed. If the database finds a match—i.e., a clear texture match—the object is known and verified as authentic. However, if no match is found during the comparison, the object is not part of the set of parts recorded by the manufacturer. This allows for the detection of a counterfeit. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2019 210 580 [0008, 0072] Zitierte Nicht-Patentliteratur

[0000] https: / / de.wikipedia.org / wiki / DataMatrix-Code

[0004] G. Levi and T. Hassner. LATCH: Learned arrangements of three patch codes. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9, 2016

[0074]

Claims

[1] Method (20) for determining whether a predetermined image section of a first image (10) occurs in a second image (12), comprising the following steps: Determining (S21) at least one image feature for a plurality of pixels within the predetermined image section; Obtaining (S23) the second image (12); Determining (S24) at least one image feature for a plurality of pixels of the second image; Comparing (S25) the image features of the second image (12) with the image features of the first image (10) to determine whether both have an identical value, wherein if at least one pair of identical image features is present, it is output that the predetermined image section is present in the second image (12), otherwise it is output that the predetermined image section is not present in the second image (12), characterized by , that the image features are calculated depending on pixel values ​​of neighboring pixels, and that in the second image a rotationally symmetric surface is determined and this area is transformed into a quadrangular area and that the image features of the pixels of the second image are calculated from pixels within the transformed area. [2] Method according to claim 1, characterized by that after the step of determining the image features of the first image (10), an assignment step (S22) is carried out, which assigns to each of the image features of the first image (10) a relative position of the pixel for which the respective first image feature was determined with respect to a reference position of the predetermined image section, wherein in the step of comparing (S25), if the values ​​of the image features of the first and second images are identical, the associated relative position of the image feature of the first image is taken and a pixel position of the pixel of the second image for which the identical image feature is present is stored relatively shifted by said relative position, wherein a pixel-by-pixel aggregation of the shifted, stored positions takes place, wherein if a plurality of the aggregated positions are present at a substantially identical pixel position, it is output that the predetermined image section is present in the second image, otherwise it is output that the predetermined image section is not present in the second image. [3] Method according to claim 1 or 2, characterized bythat a plurality of different image sections of the first image are predetermined, wherein the step of comparing (S25) is carried out for each of the image sections, wherein an overall confidence measure (qTotal=1M∑i=1Mqi) is determined by accumulation over confidences. [4] Method according to claim 3, characterized by in that the majority of the image sections have a predetermined arrangement (11) within the first image (10) in relation to one another, wherein in the step of comparing, in addition to the aggregated positions, it is determined which arrangement (13) the aggregated positions have in relation to one another and wherein the two arrangements (11, 13) are compared with one another, wherein the output as to whether the image sections appear in the second image is additionally dependent on whether the two arrangements are substantially the same. [5] Method according to one of the preceding claims 3 or 4, characterized bythat a number and / or positions and / or dimensions of the image sections are determined, wherein initially random or predetermined positions are selected, wherein the method (20) is carried out for all image sections for a plurality of first images and with at least one second image, wherein for each pair of first and second images a confidence (qTotal=1M∑i=1Mqi) is determined, whereby the number, positions and / or dimensions are changed by means of an optimization procedure so that the confidence is as high as possible if the first and second image show the same object and otherwise the confidence is as small as possible. [6] Use of the method according to one of the preceding claims for checking whether a surface image of an object belongs to a given object, wherein the first and second images depict a surface of the same object, wherein if at least one image section occurs in the second image, it is output that the surfaces of the first and second images are the same. [7] Use of the method according to one of the preceding claims 1 to 5, wherein the method is used to verify the authenticity of a product, wherein the first and second images depict a surface of the same product, in particular an object, wherein if at least one image section occurs in the second image, it is output that the surfaces of the first and second images are the same. [8] Device which is arranged to carry out the method according to one of the preceding claims 1 to 5. [9] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claims 1 to 5. [10] A machine-readable storage medium on which the computer program according to claim 9 is stored.

Citation Information

Patent Citations

  • Method for determining the position of a first image area in a correspondence image, SoC and control device and system for carrying out the method, as well as a computer program

    DE102019210580A1