Method and apparatus for determining whether predetermined image clipping of first image appears in second image

The method addresses the inefficiencies of translation-variant image matching by using translation-invariant feature comparison, enabling reliable identification of objects with random textures without complex alignment, enhancing traceability and brand protection.

JP2024170318APending Publication Date: 2024-12-06ROBERT BOSCH GMBH

Patent Information

Application Number
JP2024084759
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-05-24
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing image matching methods, particularly those using fingerprint techniques, lack translation invariance, leading to incorrect results when the Region of Interest (ROI) is shifted, and require costly and complex alignment processes, making them inefficient for objects without defined edges or reproducible poses.

Method used

A method that identifies image features within a predetermined image cutout, using translation-invariant calculations to compare image features across images, allowing for partial overlap and reducing computational effort by limiting search areas, suitable for objects with random surface textures.

Benefits of technology

The method provides reliable and efficient identification of objects with random surface textures, eliminating the need for costly alignment and ensuring accurate matching despite shifts or rotations, even for objects without defined edges, thus enhancing traceability and brand protection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024170318000001_ABST
    Figure 2024170318000001_ABST
Patent Text Reader

Abstract

To provide a method for determining whether a predetermined image clipping of a first image appears in a second image.SOLUTION: A method comprises the steps of: identifying at least one image feature for each of a plurality of pixels within a predetermined image clipping (S21); identifying at least one image feature for each of a plurality of pixels of a second image (S24); and comparing the identified image features of the second image with the identified image features of the first image to determine whether both images have the same value (S25). The method is characterized in that a rotationally symmetric surface is determined from the second image, the surface is transformed into a rectangular surface, and the image features of the pixels of the second image are calculated from pixels within the transformed surface.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method, apparatus, computer program, and machine-readable storage medium for determining whether a predefined image crop of a first image appears in a second image. [Background technology]

[0002] Prior Art In the production of subcontracted components, for example for the automotive industry, complete traceability of the individual components used therein is increasingly required and in some cases is even required by law or standards.

[0003] Such traceability makes it possible, for example, to deduce the cause of malfunctions more quickly and more accurately: for example, it makes it possible to identify the machine that produced them and the process parameters that were used, and it is also possible to identify further subcontracted parts that are responsible for the error, or to narrow it down and limit recall actions to the end product that is really affected.

[0004] This traceability in production is achieved in practice by: a) marking each individual part, for example by laser engraving, needle marking, printing or as a seal, often today with DMC (Data Matrix Code, see https: / / de.wikipedia.org / wiki / DataMatrix-Code); b. Tracking individual parts based on their individual surface textures without marking This can be guaranteed by

[0005] In both cases, to automatically read the DMC or to detect the texture, a sensor, e.g. a camera, detects the part to be tracked at a given point on the surface. This texture is then compared with previously detected textures at approximately the same point and stored (e.g. in a database using a unique index), more specifically for all the parts under consideration. In other words, in the case of the database matching described above, the current "query" record is compared with previously detected reference records.

[0006] The texture comparison gives a high match if the parts are identical, and a very low match if they are not. In this case, for example, a comparison is made with 100, 10,000 or 1 million parts. Usually, this comparison must be made in a cycle-safe manner, i.e. reasonably quickly so as not to slow down the production cycle, for example 10,000 comparisons in a maximum of 200 ms.

[0007] Tracking without marking has several advantages. i. Reduced effort, cost and investment. For example, a laser engraving machine is much more expensive than a camera, especially in purchase cost, including lighting for texture detection. ii. Laser engraving of DMC produces smoke, i.e. small spalling from the surface, which can contaminate components (e.g. nozzles) and thus damage their functionality, whereas tracing without marking is perfectly clean. iii. Many components are too small or do not have a suitable area for DMC deposition, whereas for texture detection, a minimum area of ​​1 mm 2 That alone is enough. vi.DMC can be read and used by unauthorized persons with harmful intent, whereas in the case of textures this is not possible, since only the manufacturer itself knows the texture mapping, which makes the method more secure. v. Protected against counterfeiting: a counterfeit component can be given a copied DMC without incident, whereas cloning the random surface texture of a component is practically physically impossible.

[0008] DE 102019210580 A1 discloses a method for verifying the authenticity of an object, in which a user, for example, takes a picture of the object, for example a banknote, via a smartphone (app) and immediately thereafter receives a corresponding acknowledgement from the app if it is authentic. The texture comparison is performed, for example, in the cloud, where reference textures are stored in a database and can be used for the comparison. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] DE 102019210580 Summary of the Invention [Problem to be solved by the invention]

[0010] Advantages of the invention The first and most important advantage of the proposed method is its translation-invariance. This means that the identification is invariant (independent) to the translation of the query object in the camera image. In other words, the identification will give the same result regardless of where the ROI (Region of Interest) is shifted in the second image, provided that it naturally completely contains the correspondence with the first image region. The translation-invariance applies to the two axes (x,y) of the camera image. The need in the prior art to be able to reproduce the pose of the reference ROI in the query image is omitted. That is to say, the alignment step required by fragile fingerprinting methods can be omitted in the present method.

[0011] A second advantage is scalability in the sense that the user has the possibility to limit the allowed (x,y) translations. Basically, the method is still able to identify an object even if there is still only a small overlap between the reference record and the query record for that object. The user can then limit the translation area (in the sense of the search area) to the required extent, for example to 20 pixels each to the left, right, up and down. That is to say, in this example, the translation vectors may be located within an area of ​​size 41x41 pixels. The advantage of this limitation is that the computational effort (computation time) is saved, since it increases with the area of ​​the translation area. However, the computational effort (computation time) increases much more slowly than in the known fingerprinting methods.

[0012] These advantages regarding translation make it possible to apply the proposed method to object types that could not be managed by fingerprinting methods so far, e.g. for rough cast parts or for parts where there are no available visual references (edges, corners, holes, etc.) that allow proper alignment (a pre-processing step to uniformly align the query images).

[0013] A third advantage of the proposed method is that its similarity measure (described further below) has in practice proven to be much more discriminatory than the Hamming distance measure of fingerprinting.

[0014] A fourth advantage is that the proposed method can cope with small image dimensions in practice: even for very difficult object types, an image dimension of, for example, 400x400 pixels is already more than sufficient; in many cases, an image dimension of about 50x50 pixels is already sufficient in practice for reliable identification. [Means for solving the problem]

[0015] Disclosure of the Invention In a first aspect, the invention relates to a computer-implemented method for identifying whether a predefined image crop of a first image appears in a second image, the image crop being pre-selected manually or determined by other methods, in particular an image crop that is relatively small relative to the first image, particularly advantageously including an image crop of 50x50 pixels to 400x400 pixels of the first image.

[0016] The method starts by determining image features for a number of pixels within a given image crop. The image features are determined according to a given calculation rule, which calculates the image features depending on the pixel values ​​of the neighboring pixels. For example, a surrounding of 3x3 pixels to 127x127 pixels including neighboring pixels around the respective pixel can be considered for the image feature. The calculation rule outputs a unique value, and thus the image feature is unique, i.e. the image feature characterizes the considered pixel arrangement of the considered pixel including its surroundings. In this case, the calculation rule is selected such that the unambiguous image feature is obtained with a high probability even in the case of at least slight changes in pixel values, for example due to sensor noise or lighting differences. That is, the calculation rule does not have to determine a unique value for each pixel combination, but preferably provides unambiguity with a high probability. Preferably, the calculation rule is translation invariant, at least within a small range.

[0017] This is followed by receiving a second image, in particular a crop (English: ROI) of the second image that is smaller than the second image but at least as large as the image crop of the first image.

[0018] This is followed by identifying image features for a number of pixels of the second image or within the ROI, where the image features of the first image and the image features of the second image are calculated in the same way, i.e. similar calculation rules are used for the image features.

[0019] This is followed by a comparison of the identified image features of the second image with the identified image features of the first image, preferably taking into account not only the values ​​of the image features but also their relative arrangement with respect to one another in the respective images. If a group of at least two non-identical image features in the second image is found that is equally arranged or nearly equally arranged with a group of non-identical image features of the same value in the first image, then an output is made that the predetermined image crop is present in the second image, otherwise an output is made that the predetermined image crop is not present in the second image.

[0020] The particularity of the method is that a rotationally symmetric surface, in particular an arc-shaped or circular surface, is determined from the second image and in particular from the first image, which is transformed into a quadrangular surface, in particular into a rectangular surface, and image characteristics of the pixels of the second image and in particular of the pixels of the first image are calculated from the pixels inside the transformed surface. The rotationally symmetric / arc-shaped surface can be transformed into a polar version with predefined dimensions of height x width by a given center point, a "start" radius and an "end" radius, and preferably a start / end angle. It should be noted that a correspondingly predefined image crop originates from the transformed surface of the first image.

[0021] It is conceivable that adjacent pixels of the rotationally symmetric surface, which are now spaced apart from one another on the quadrilateral surface and are located at opposite ends of the quadrilateral surface, can be included in the calculation rules when calculating the image characteristics of these originally adjacent pixels, in other words, the ends of the quadrilateral surface can be supplemented, for example, by copies of the pixels at the opposite ends.

[0022] After the step of identifying image features of the first image, it is proposed to carry out a step of associating with each of these image features a relative position of the pixel at which the respective first image feature is identified with respect to a reference position of the predefined image crop. The reference position may be the center of the image crop. In the step of comparing, if the values ​​of the image features of the first image and of the second image are identical, the associated relative position of the image feature of the first image is adopted and the pixel positions of the pixels of the second image at which the identical image feature is present are relatively shifted around said relative position and stored in particular at this shifted pixel position in the voting weight matrix. The shifted and stored positions are then aggregated pixel by pixel and then, if the majority of the positions outputted in terms of the confidence measure are present at substantially equivalent pixel positions of the second image, it is outputted that the predefined image crop is present in the second image, otherwise it is outputted that the predefined image crop is not present in the second image.

[0023] Substantially equivalent pixel positions may be understood to be deviations of ±2, 3, or 4 pixels or more.

[0024] This may optionally be followed by an outputting step, in which the pixel location with the highest count after aggregation is output as the location of the image crop in the second image.

[0025] Optionally, to facilitate evaluation, the aggregation can be smoothed, thereby achieving a clear maximum at a unique location while at the same time suppressing unimportant secondary maxima.

[0026] Preferably, the image features are used as addresses in a look-up table, with each address being stored with an associated relative position. Optionally, the look-up table further comprises a weighting in addition to the relative position, which weighting varies, in particular decreases, as the distance from the image feature position to the reference position increases. The weighting can additionally be used during aggregation to perform this aggregation in a weighted manner.

[0027] If there are multiple image crops, each of the above method steps is performed for each different image crop. Preferably, the image crops are evenly distributed over the object texture in the first image, for example evenly distributed over a circle.

[0028] It is further proposed that the image cutouts have a predefined arrangement relative to one another within the first image, and that in the comparing step, in addition to the stored positions, it is further proposed to determine how these positions are arranged relative to one another and to compare these two arrangements with one another. This arrangement may be given by a predefined distance between the positions or by a positional relationship pattern. When positional relationship patterns are compared, preferably only the structure of the patterns is compared, without taking into account the alignment of the patterns. In this case, it is advantageous that an additional independent criterion of the positional relationship arrangement effectively provides further confirmation of the information whether the imaged objects on the images are identical.

[0029] It should be noted that the step of determining image characteristics of the pixels of the second image is preferably performed in a computing unit immediately after the second image is detected and transmitted to a server, which returns a signal depending on the comparison. Preferably, the image characteristics of the image crop are already determined or transmitted to the cloud, where a comparison with the received image characteristics is simply performed. Alternatively to the cloud, the method may also be performed locally.

[0030] Preferably, in the first aspect of the present invention, the first image and the second image are images of the surface of the same workpiece.

[0031] The first and second images may have been taken by one or more identical cameras. No special requirements need to be placed on the cameras for detecting these two images, a low image resolution is already sufficient. The imagers should be constructed in both cases as similar as possible with regard to the choice of objective lens, the distance to the selected surfaces of the object, the focus on these surfaces, the type and arrangement of the illumination. The exposure time and the illumination intensity should be selected so that the respective images do not have a significant motion blur and are neither too dark nor too scattered. If it is not possible to use two identical camera devices, the differences can be compensated for by image processing afterwards. For example, if the second camera has a different distance or focal length or pixel format (pixel pitch) than the first camera, the resulting scaling differences can be compensated for via image scaling so that the considered surfaces of the object are subsequently each present with approximately the same size (in pixels).

[0032] In a further aspect, the invention relates to the use of one of the methods of the first aspect for checking whether a surface record of an object belongs to a given object. Preferably, the query object is compared with all reference objects. The best match (highest overall confidence value) is then identified. If this value is high enough (above a threshold), a match is output. But if the best overall confidence value is not high enough, no match has been found. The query object is either counterfeit or has still not been detected as a reference.

[0033] In a further aspect, the present invention relates to the use of one of the methods of the first aspect for verifying the authenticity of a product.

[0034] In further aspects the invention relates to an apparatus and a computer program respectively adapted to carry out the above method, and to a machine-readable storage medium on which this computer program is stored.

[0035] In the following, embodiments of the invention will be described in more detail with reference to the accompanying drawings. [Brief description of the drawings]

[0036] [Figure 1] 3A-3C show schematic images of a manufactured component, each of which comprises a number of image areas AL in a raster-like arrangement. [Diagram 2] FIG. 2 illustrates a schematic diagram of an embodiment of a method for re-identifying an image region. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] There are already several methods for marking-free tracking based on surface texture, for example the so-called Track & Trace Fingerprint from the Fraunhofer Institute for Physical Measurement Technology. This technology is based on the fact that many semi-finished products or components have microscopically individually characteristic surface structures or color textures. Certain areas of the components are selected and each photographed at high resolution by an industrial camera. From the image recording with the respective specific structures and their positions, a numerical identifier is calculated and associated with an ID. This pairing is stored in a database together with further data, such as measurement or production data. For later identification, the entire process is repeated and the ID of the component and further individual features are returned by data matching.

[0038] The "computed numerical identifier" described above will be referred to below as a "fingerprint." This fingerprint may be thought of as a number with many digits, or it may be thought of as synonymous with a long bit sequence.

[0039] In the database matching described above, the current fingerprint is compared with previously detected reference fingerprints and the degree of difference between them is determined. Ideally, exactly one reference fingerprint is found that is only slightly different from the query fingerprint, while comparison with all other reference fingerprints shows large differences. If and only if this is the case, the identification is considered successful.

[0040] The dissimilarity between a physical reference object and a physical query object is typically determined from the fingerprint pair alone, e.g. as the Hamming distance of the two bit sequences, or according to the definition of a further distance measure.

[0041] In this respect, the present invention is particularly distinct from the prior art.

[0042] Methods known from the prior art for traceability and brand protection are based on forming so-called fingerprints describing the individual texture properties for an object's ROI (Region of Interest) as bit strings of fixed length, which can be, for example, several thousand bits.

[0043] In that case, the texture comparison is performed as a fingerprint comparison: for example, the Hamming distance between the fingerprint of the reference object and the fingerprint of the query object is formed, i.e. they are compared bit by bit and the differences are counted. The smaller the Hamming distance (ideally 0), the more similar the objects are.

[0044] Normalizing the Hamming distance with the length of the bit string gives a similarity value between 0 and 1, with practical values ​​only utilizing the range from 0 to about 0.5, since if bit strings are statistically independent, then only about half of the bits will differ.

[0045] For example, comparing fingerprints via Hamming distance has the advantage of speed, ie, a large number of comparisons can be performed in a short amount of time.

[0046] However, fingerprint comparison via the Hamming distance also has a significant drawback, since the results of the fingerprint comparison are not translation invariant: for example, if a fingerprint for a query object is generated in a different ROI than for the reference object, for example shifted by 20 pixels, the query object will no longer have a very good similarity to the reference fingerprint.

[0047] That is, in this case, texture comparison will give erroneous results: the texture image pair will be erroneously classified as dissimilar, despite being the same object.

[0048] This adverse behavior due to the lack of translation invariance of fingerprinting means that in practice it is usually necessary to incur a large additional cost to make the pose of the ROI in both the reference and the query images as identical as possible, since deviations of even a few pixels are no longer acceptable.

[0049] For some object types, it is also possible to mechanically move the object into a reproducible pose relative to the camera based on the geometry of the part. This creates additional effort (robot gripper arms to grasp the object, place it on a suitable template in front of the camera, re-grasp the object, and return the object, time, cost, fragility). For other object types, these geometric conditions are not given and visually salient features of the object type must be used to make the ROI reproducible, e.g. holes, grooves, corners, edge poses, etc. This also creates additional effort (adjustment of image analysis algorithms to the component, application by an expert).

[0050] However, there are object types for which neither one nor the other of the above techniques works to reproducibly rediscover the ROI.

[0051] This applies, for example, to objects of a relatively rough nature, such as metal casting parts that do not have defined edges, but rather have edges that are significantly beveled and rounded to allow for removal from the casting mold, and that potentially have random casting burrs that create additional difficulties.

[0052] Even for flat object types such as metal sheets, paper, cardboard, fiber-plastic plates, etc., reproducibility of the ROI pose can be difficult or impossible, especially if edges are bent, folded, punched or cut during processing, thus losing possible visual references.

[0053] It is certainly possible to capture relatively small uncertainties in the ROI pose by forming multiple fingerprints with slight shifts in the ROI position (e.g., pixel by pixel). However, this increases the effort for fingerprint formation and comparison correspondingly many-fold, and in many cases even quadratically, since there is uncertainty in both axes (x,y). It is practically impossible to accommodate relatively large uncertainties in the ROI pose.

[0054] For fingerprinting to work in operation, both steps must be successful at all times: the registration step (also called the matching or alignment step) and the fingerprint comparison step.

[0055] A particular difficulty is that the registration steps that must be performed for alignment of the ROI at each camera station (a first station for initial detection and several further stations for identification) must always lead to identical results, which in practice can hardly be guaranteed, especially during continuous operation.

[0056] Fingerprinting is therefore regarded as being in practice vulnerable and is accordingly unpopular with users: for example, a slight deviation of the camera with respect to the mounting template means that the components at the identification station are no longer identifiable and therefore production is halted.

[0057] In what follows, we introduce a confidence measure to assess our confidence in the quality of the found correspondences.

[0058] Confidence measure q B is mathematically B= (p1 / p2)-1, where p1 is the height of the best peak and p2 is the height of the second best peak. Peak here means the value of a global or local maximum in the voting weight matrix or in a further processed form of the voting weight matrix.

[0059] A preferred further processing of the voting weight matrix consists in smoothing the voting weight matrix after the accumulation (collection of the voting weights) is completed. A suitable smoothing filter is for example a two-dimensional Gaussian shaped smoothing filter.

[0060] Since p1 ≥ p2 is always satisfied, q B ≧0.

[0061] Confidence measure q B is suitable for determining whether the search for a correspondence has been successful, for example, already from an individual image comparison (one-to-one comparison: one query image or query ROI for one reference image region). If a correspondence has been found, q B is large (e.g., q B =11), otherwise, q B is small (e.g., q B =0.1).

[0062] An alternative embodiment of the confidence measure may omit the subtraction of 1.

[0063] An even easier alternative reliability measure, q A is a function q with an optional constant c A = c p1, i.e. the height of the best peak is used as the confidence measure.

[0064] By this we also mean the value of the global maximum in the voting weight matrix, or in a further processed form of the voting weight matrix (e.g., following smoothing filtering). A To specify a range of values ​​for, e.g., 0≦q AAvailable to specify ≦1.

[0065] Confidence measure q A has the advantage that the identification of the second peak p2 can be omitted, which means a saving of calculation operations, but on the other hand it also omits a comparison value for normalization.

[0066] However, this apparent disadvantage is not important in the 1 vs. N comparison case (one query image against multiple (N) reference images, as discussed further below), because in this case, all lower q A For N-1 non-hits with a high q A Since there is one hit characterized by the value, this information is entirely sufficient to determine the winner, and therefore the levels of each second best peak in the same query image become optional information.

[0067] Therefore, in the following, we use the simpler confidence measure q A Let be the preferred confidence measure.

[0068] In order for the invention to function reliably, a camera is preferably used which detects a two-dimensional image of the (partial) surface of an object in the form of grayscale or multidimensional (color) values ​​(so-called texture detection), which image is as reproducible as possible from camera to camera. In other words, it is desirable for the images to look as similar as possible. In this case, easily correctable differences, which can be corrected for example by rotating, scaling, shifting or luminance matching of the image content, are not an issue.

[0069] It is advisable that both recording devices are as similar as possible, or at least coordinated with one another, so that the image content of two recordings of the same object is as similar as possible in this sense. This applies in particular to the lighting.

[0070] In the case of objects with a non-shiny surface, this is relatively easy to achieve, since the incoming light is diffused and scattered, so that the image of the object changes very little with small changes in the illumination position. In the case of objects with a strongly reflective surface, for example polished or brushed steel followed by chromium plating, the image depends strongly on the position of the light source, especially if the light source is point-like. In order that both capture devices still provide reproducibly similar images of the object, it is highly recommended to choose illumination such that the light is incident on the surface as uniformly as possible from all directions (diffuse illumination). This is achieved, for example, by a dome illumination near the top of the object, in which case the inner surface of a matt white hemisphere is illuminated so that indirect illumination is as omnidirectional as possible. In this case, the camera looks into the hemisphere through a hole located inside the hemisphere or in the region of the hemisphere's axis of symmetry.

[0071] Especially in the case of such difficult surface types, it may be worthwhile to obtain the texture image from several successive shots with the same camera, each under different illumination. For example, four shots can be taken in short succession, in which only one quadrant of a ring-shaped or dome-shaped diffuse illumination is switched on in each of the four shots. Suitable known methods are, for example, Shape from Shading or Photometric Stereo.

[0072] For example, in Shape from Shading, a so-called texture image describing the reflectance of the surface and a curvature image describing the local curvature are calculated. Both images, either independently of each other or in combination, are suitable for the present invention, since both provide a reproducible image of the object, each of which has its dependence on the illumination calculated.

[0073] Different methods for refining image regions from a first image in a second image are known, for example from DE 10 2019 210 580 A1.

[0074] One of these methods first defines a small outline of the image crop, for example a square or a circle or a polygonal outline.

[0075] Inside this contour, there are several feature points, i.e. image features formed from their respective environments, each represented by, for example, 16 bits. Such image features are more or less compact representations of local image content, i.e., the larger overall image area contributes to the completeness of the formed feature. To calculate the image feature for a given pixel, the Latch Descriptor by G. Levi and T. Hassner. LATCH: Learned arrangements of three patch codes. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9, 2016. can be used. However, it should be noted that there are many further known methods for calculating these features.

[0076] In this example, the square is 50x50 pixels in size and contains 2500 feature points, in other words the feature density is here one feature per pixel, however the feature density may be higher or lower than this, for example four features per pixel or one feature per nine pixels.

[0077] Each feature value (e.g., with a word length of 16 bits) is used as an address into a Lookup-Table, where the Lookup-Table is filled in with the feature's position in the image relative to a reference position in the image, and optionally with further information, e.g. weights.

[0078] Once at the latest all 2500 feature points have been entered into the table (in most cases a smaller number is already sufficient), the processing of the considered image region from the first image is thereby terminated.

[0079] The second image is the image in which the position corresponding to the reference position is to be determined. In this case, the search area for this position may include the entire second image. Correspondingly, features are then generated for the entire second image. Alternatively, the search area includes only a part (ROI) of the second image. In that case, features are generated for the second image for the surface corresponding to the enlarged ROI, where this ROI is enlarged by the allowed shift, for example by 20 pixels to the left, right, up and down. For the second image, exactly the same methodology is applied for feature calculation as for the first image, or an adapted methodology is applied that appropriately takes into account changes in texture state, for example from dry and clean to oily.

[0080] In this case, prior knowledge can be used that describes, for example statistically, the change in texture in the image when transitioning from a dry state to an oily state or vice versa, for example in the form of a complex distribution density of the local contrast (or color values) between the two states.

[0081] The feature density in the second image is again one feature per pixel in this example, but may be higher or lower and may not match the feature density of the first image.

[0082] The generated features from the second image are again used to address the lookup table, but this time in a read-out format. Starting from the current feature position in the second image at the relative position determined based on the lookup table, as described above, the voting weights are output and entered into the voting weight image. This can be done for each feature in the search area, and the voting outputs in the voting weight image are collected, preferably additively or weighted additively.

[0083] In an advantageous embodiment, the size of the voting weight image corresponds approximately to the size of the search region. The voting weight image may be chosen smaller or larger. However, the voting weight image does not have to be chosen larger than the reach of the relative position at the time of the voting output. This reach depends on the size of the first image region and the pose of the reference position associated therewith, or in other words on the boundedness of the length of the 2D vector.

[0084] In the method for rediscovering image regions, if there is a correspondence in the vote weight image, a very distinctive and spatially very concentrated collection point is obtained, the pose of which corresponds to the searched position, i.e. corresponds to the reference position from the first image in the second image. For example, by evaluating the collection point in the sense of identifying the maximum position or the centroid position of the collection point, the searched corresponding position can be identified with sub-pixel accuracy if desired or necessary. Depending on the resolution of the camera, this can correspond to sub-millimeter accuracy.

[0085] The determined position information can be advantageously used in different ways. From the difference between the reference position in the first image and the found position in the second image, it can be determined how the pose between the camera and the object differs between the two camera stations. From this, statistics can be determined. The average value provides information about the dissimilarity from camera station to camera station, which in some cases is desired to eliminate. The deviation from the average value, expressed for example as a standard deviation, provides information about how the pose varies from object to object. Such position variations can be caused by changes in gripping with the robot or by differences in the shape of the parts. For example, such variations can be large in the case of cast parts that have not been deburred.

[0086] Statistics may also be useful to parameterize the search region, since, as mentioned above, it makes sense to construct a relatively small search region to reduce the computational effort, rather than fully utilizing the translation degrees of freedom, but it should not be chosen too small, in order not to miss correspondences.

[0087] If multiple correspondences are found for an image pair, inferences can be made from the alignment of the found correspondences, further improving the reliability of the classification.

[0088] In the following we consider how advantage can be gained from the optional use of multiple reference image regions.

[0089] A number of reference image regions are located in the reference image. For example, M=12 reference image regions are located relative to the reference image in such a way that they detect areas of the object surface that preferably allow distinguishability. For re-identifying individual objects, noise-like textures resulting from random processes are particularly well suited. In contrast, areas that look almost the same on both objects would not be suitable. Surfaces that are not suitable for locating reference image regions are, for example, overly scattered image regions that are so bright that the camera can no longer resolve details, or very dark image regions with a low signal-to-noise ratio, where the image signal is substantially determined by sensor noise (which of course cannot contribute to identification).

[0090] Moreover, unsuitable surfaces may be, for example, almost homogeneous image regions where the signal is almost constant or only changes locally slightly, i.e. the information does not contribute or only contributes little to the discrimination, or very smearly detected image regions located far outside the focal plane or focal plane of the camera, or image regions such as, for example, edges, corners, holes, which have no role in distinguishing objects, since they look the same in all objects. Such image regions also include the edges of shadows caused by illumination.

[0091] Furthermore, not suitable are surfaces that have been destroyed in the course of tracking an object or that have been subjected to significant changes that make tracking impossible, for example surfaces that have been scraped, polished or sandblasted, or surfaces that have been coated with an opaque paint or otherwise obscured so that they are no longer visible afterwards.

[0092] The specification of the reference image area is usually performed only once per component type after the acquisition equipment (camera, lighting, pose, image resolution, image crop) has been determined. Such a specification is usually the only parameterization step that remains necessary. The specification of the reference image area is then applied, in the usual case, to all objects of this component type. The specification of the reference image area may be performed by an expert or may be performed automatically.

[0093] For example, the specification can be carried out by an expert, more specifically using M=12 reference image areas, the centers of which are located at equal angular distances of 30° on a circle, similar to the hours on a clock face. This arrangement is suitable for example for circular or annular object surfaces. The reference image areas are preferably located in a well-focused, randomly textured surface. Edges and blurred image areas are avoided.

[0094] Although overlapping reference image regions is certainly permissible, it is desirable to avoid it, since no additional benefit is gained by the additional effort.

[0095] Global confidence measure q when using multiple image regions Ges In the simplest case, a sum or average from M confidence measures is formed, i.e., for example:

number

[0096] An advantage of this method is that it is possible to position multiple small reference image regions independently on suitable partial surfaces each with a suitable surface texture, and thus an optimized adjustment to the component type in terms of distinguishability can be achieved with little effort.

[0097] Utilizing an expert to specify the reference image regions may be disadvantageous for reasons of cost or availability.

[0098] Alternatively, it is possible to automatically optimally specify the image region. For this purpose, with a very small number of objects, e.g. N R = 10 objects, but at least N R = Training that can be handled with 2 objects is envisioned.

[0099] In the following, these N R There is one reference record (hence the subscript R) from each of the training objects, of which N Q ≦N R It is assumed that there is also one query record (index Q) from each of the

[0100] In this case, 1≦N Q ≦N R or, in other words, ideally there is one query record per object, i.e., N Q =N R However, if necessary, only one query record, namely N Q = 1 is also sufficient.

[0101] The training is carried out, for example, as follows. 1. Initially, a predetermined number M of reference image regions are positioned, for example randomly or regularly, for example as a tiling. 2. This positioning is N R Then, N QQuery images and N R The method is applied to compare the image pairwise with N reference images. R ×N Q Similarity values ​​are generated, and these similarity values ​​are R ×N Q , where the column index corresponds to the query number and the row index corresponds to the reference number. N R =N Q and the order is the same, then there is a correct correspondence on the main diagonal. 3. The similarity matrix is ​​evaluated. The goal is to optimize the similarity matrix, achieving a maximum for correct matching in each column (usually on the main diagonal), while making all other values ​​in that column as small as possible. This measure may be, for example, the quotient or difference of the highest similarity to the second highest similarity (in each column), averaged over all columns. This gives an overall evaluation number for the positioning setup, and it is essential to maximize this overall evaluation number. 4. If the total number of evaluations has reached a maximum or is above a threshold, the optimization is complete and therefore the identified parameters have also been optimized. If not, continue. 5. The setup is changed, including changing the position of the individual reference image areas (i.e. their orientation within the image) and / or their dimensions (e.g. width, height, aspect ratio, angle in case of rectangular or elliptical shape) and / or the number of reference image areas.

[0102] Again, note that overlapping reference image regions is avoided as much as possible.

[0103] Of course, common methods in optimization can be used to accelerate this training or to guide it towards a global optimum, for example gradients can be determined to identify the parameters that have the greatest impact on the overall number of evaluations and prioritize their modification.

[0104] It is advantageous to include the computational effort in the overall evaluation number (the smaller the computational effort, the better the evaluation number), which automatically results in the reference image regions not being unnecessarily large in number and size, nor overlapping with each other.

[0105] Objects are often constructed with substantial rotational symmetry, and it is often not possible to always reproducibly position the object in front of the camera at the same angle (around the axis of rotational symmetry). For example, a lateral cutout would certainly provide unambiguity and thus assistance, but such assistance is not provided for most object types.

[0106] For such and similar objects which are non-unique in this sense with respect to rotations about an axis, the invention has a distinct advantage over the prior art, since it allows the translational invariance of the object in two directions to be transformed into a tangential invariance and an axial invariance, where the tangential invariance, sometimes called rotational invariance, is of particular importance.

[0107] For this purpose, it is proposed to digitally develop a cutout from the image, for example a torus-shaped surface, for which two radii of the disk in the image and the location of the disk's center point can be specified, which should preferably coincide at least approximately with the piercing point of the axis of rotational symmetry in the image.

[0108] The parameters for cutting the disk are known beforehand or can be determined from the image. In particular, it may be useful to determine the center point anew for each shot, for example based on an estimate of the center of symmetry, if this center point changes. In this case, a rough estimate, for example an accurate estimate of a few pixels, is already sufficient.

[0109] When digitally unwrapping a disk, it is cut open and unwrapped into squares, i.e. the pixel values ​​(gray values, color values) are warped accordingly.

[0110] During the unfolding, it should be noted that not only can surfaces such as circular arcs be unfolded, but that the surface to be unfolded can of course also be an inner or outer cylinder, or an inner (frustum) or outer cone (frustum). Even drum-, barbell- or wave-like shapes are conceivable. Thus, in general, the unfolding can be applied to rotationally symmetric surfaces.

[0111] At the cutting line, it can be assumed that the rectangle is continued periodically to the left and to the right, respectively, since both ends are paired. Correspondingly, all image processing operations and further processing steps can be continued without interruption beyond this cutting line.

[0112] After unfolding the disk-shaped image content into a rectangular image, the method can then be applied as usual: each unfolded query image is compared with the unfolded reference image.

[0113] Now, if the above principle of periodic continuation is consistently followed, the result of the identification according to the invention will be independent of where on the annulus the radial cross section is located.

[0114] If we assume that the round component is in an unknown angular orientation, this unknown angular orientation will result in an unknown horizontal translation due to the unfolding. However, the algorithm is invariant to this, which means that the result is independent of the horizontal translation. That is, the object can be identified in its unfolded form despite the unknown angular orientation. Thus, the translation invariant method will result in a rotation invariant method due to the unfolding.

[0115] In the unfolded image, the vertical translation invariance is also given, which means that vertically shifted correspondences are also found. In this case, the vertical translation invariance is mapped to the radial invariance. This radial invariance is also very useful, since it allows to compensate for incorrectly positioned center points. In other words, even if the estimation of the symmetry center is erroneous, for example shifted by 5 pixels, and the unfolding of the disk is performed around a correspondingly erroneous center point, this is not a problem, since it is still possible to find the (now vertically shifted) correspondences. This gives the method a further and extremely enormous advantage of robustness.

[0116] In summary, this means, in other words, that in order to identify a substantially rotationally symmetric object, in practice the angular orientation of the substantially rotationally symmetric object in the image recording may be arbitrary and unknown, and it is sufficient to only roughly know or estimate the orientation of the rotation axis or centre point.

[0117] This important advantage over the prior art is achieved by the given two-dimensional translation invariance of the method, which can be transformed by expansion into rotational and radial invariance.

[0118] Radial invariance is not true invariance in the mathematical sense, since the unfolding is a nonlinear mapping with scaling factors that depend on the respective radius. Errors in the orientation of the center points are correspondingly transformed into scaling errors. However, this apparent problem turns out to be negligibly small in practice and therefore unimportant.

[0119] As soon as multiple reference image regions per image are used, an optional additional means for further increasing confidence in the identification is provided.

[0120] Because the M reference image regions are present in a particular arrangement in the reference image, one can hope that the correct correspondences that can be found for them will also be present in the query image in the same or similar arrangement, which can be checked and thus an additional or alternative quality measure can be formed.

[0121] If the configuration meets this expectation, the confidence is high, and if it deviates from this expectation, the confidence drops accordingly.

[0122] If it is not possible to find all M possible correspondences, for example because the object is partially damaged, at least a residual portion of these correspondences should still exist in the expected partial configuration, and therefore this should still yield a relatively high degree of confidence.

[0123] This is illustrated in Fig. 1. On a reference object 10, M=12 image regions, here marked with letters A to L, are specified, which image regions are to be used for texture-based identification. The center points of each of these image regions are in a pre-specified arrangement 11, which is also illustrated graphically next to the right of the image. In this case, it is a grid-like arrangement with a rectangular grid with constant grid spacing (in both dimensions). Here, out of the 4 x 4 possible grid locations, 4 corner positions are not used.

[0124] Other arrangements of the image regions are contemplated, for example on a circle, or in other geometric shapes.

[0125] At the time of the interrogation recording, the same object 12 has in the meantime suffered cosmetic damage, which makes identifiability difficult. Further factors, such as changes in the photographing conditions, may also act to introduce additional difficulties.

[0126] All this leads to some correspondences, e.g. G, never being rediscovered, and to a significant number of correspondences, e.g. F, having very low confidence, or, like B, having a greatly reduced confidence.

[0127] In particular, for such difficult cases (which may be significantly more difficult than those presented here), a measure is proposed to introduce additional testing, in which the consistency of the placement is examined and additional evaluation metrics are derived from this.

[0128] This is illustrated in the present example. The found correspondences E and I are considered here to be particularly strong. For example, the positions of these correspondences can then be used as anchor points to anchor the known configuration, in this case the raster, accordingly as shown in the configuration 13 on the right. This results in a slight rotation of the raster to the right here (since the object has been rotated accordingly in the query record). A slight scaling (shrinkage or enlargement) of the raster may also be required here.

[0129] Starting from the raster thus anchored, all remaining expected corresponding positions can be identified, for example the corresponding position of B. If this correspondence is indeed found within a circumference centered on this expected position, this should favorably affect the combined confidence, i.e. increase the confidence. The circumference takes into account, for example, the expected measurement accuracy, image distortion, numerical inaccuracies and error propagation. The circumference can have a fixed or variable radius. Optionally, the circumference increases with the distance to the anchor point, which can also be seen in the arrangement 13 where multiple circumferences have different diameters. Optionally, the closer the anchor points are located to each other (reduced leverage, not shown here), the more the circumference increases.

[0130] The circumference should not be taken literally, i.e. it does not have to be a circular area, but can have other shapes, in particular the shape of the ellipse is important because it is the shape of the ellipse that is obtained from the error calculation.

[0131] In that case, the calculation of the combined confidence can be performed, for example, as follows: 1. Select two of the found correspondences, say E and I, and anchor a raster (here synonymous with expected alignment) at the positions of these correspondences, more precisely by appropriate shifting, scaling and rotation of the raster. 2. Check all further discovered correspondences A-D, F-H, J-L to see if they fit the anchored raster, e.g. lie within their respective expected circumferences. 3. In the simplest case, count the fulfillment of expectations. In this case, we can generate a new confidence measure, here a count value between 0 and M-2=10, not counting the anchor points themselves.

[0132] In an improved variant, instead of counting, each q A or q B The M individual confidence values ​​identified according to are summed up, more specifically, again only for correspondences located within the respective circumferences. Optionally, the individual confidence values ​​of the anchor points can also be summed up.

[0133] The formula for this is, for example,

number

[0134] In a further improved variant, the circumference criterion is taken to be more flexible, i.e. not as a yes / no decision (located in the circumference or not), but by an evaluation of the distance from each found corresponding position to each expected position, e.g. a maximum value for a perfect match, a medium value for a slightly larger distance, and about 0 for a too large distance.

[0135] Therefore, this equation is: q Ges However, instead of using a binary value, we use a scalar-valued weighting factor wi is used.

[0136] w i For example, a two-dimensional correlated or uncorrelated Gaussian density, each with its maximum at the expected position, or a scan of a two-dimensional cone-shaped density function can be used as the value of w. The number of evaluations w obtained by scanning the density function at the location of the found correspondences is i Let each individual reliability value q i (e.g., peak height). These products, here up to M-2=10, are added back together. Optionally, anchor points can also be taken into the sum together, preferably with an exact match evaluation for each.

[0137] Optionally, the last-listed step can be repeated with all other possible anchor points, for example with anchor points C and L as follows: the further away the two anchor points are from each other, the more leverage there is to anchor the raster, which is advantageous.

[0138] These permutations avoid the risk of incorrect anchoring resulting in an inappropriately bad global assessment due to incorrect correspondences. Thus, all global confidences q obtained through permutations are Ges Of these, the overall confidence level which preferably achieves the highest value is subsequently used, since it is at this overall confidence level that the underlying mooring was correct is assumed earliest.

[0139] In the example considered here (with M=12 image regions), it is possible without any problem to consider the complete amount of possible rearrangements. If one wishes to consider only a subset of the rearrangements, it is advantageous to give preference to anchor points that have a high individual confidence value and at the same time a long leverage.

[0140] Up to now, only the case with two anchor points has been considered, but even with two anchor points, the rotation and scaling adjustment between the reference image and the query image is possible. However, it is also possible to work with other numbers of anchor points, for example three or four anchor points. For example, this even allows and even allows different camera views to be allowed and taken into account with respect to the object surface. This gives the user more freedom when setting up the installation. For example, it is then no longer necessary to pay attention to the line of sight of the camera at each station being oriented perpendicular to the object surface, but also oblique line of sight angles and even handheld camera operation is possible.

[0141] FIG. 2 shows a flow chart of a method for re-recognizing an image region.

[0142] The method begins by identifying image features for a number of pixels within a given image crop (S21).

[0143] Optionally, a matching step S22 is performed in which each of the image features is matched with the relative position of the pixel at which the respective first image feature is identified relative to a reference position of the predetermined image crop.

[0144] This is followed by receiving a second image (S23), determining rotationally symmetric surfaces from the second image and transforming the surfaces into quadrilateral surfaces (S23), and identifying image features for a plurality of pixels of the second image (S24), which transformation has also been performed on the first image as described above.

[0145] This is followed by a comparison (S25) of the identified image features of the second image with the identified image features of the first image whether they both have the same value. Then, if there is at least one pair of identical image features, it is output that the predetermined image crop is present in the second image, otherwise it is output that the predetermined image crop is not present in the second image. It should be noted that, for example in the case of image features that are only limitedly unique, there should preferably be at least two or more pairs of identical image features. Preferably, the pairs of identical image features are present at equal positions in the voting weight matrix. If step S24 has been performed, in the comparing step (S25), if the values ​​of the image features of the first image and the image features of the second image are identical, the associated relative positions of the image features of the first image can be adopted, and the pixel positions of the pixels of the second image where the identical image features are present can be stored with a relative shift around said relative positions. As mentioned above, this determined position can be interpreted as a voting weight that is summed up in the voting weight matrix (voting image) for each pixel. The voting weight matrix can be used to determine the location of the image crop within the second image.

[0146] The two most important fields of use for the method according to FIG.

[0147] Based on the individual surface texture, components can for example be tracked without marking in industrial production (traceability). Brands are protected by using the individual surface texture of products to re-identify originals from your own production and to distinguish / detect third-party products or counterfeits.

[0148] In both cases, the naturally occurring texture of the object's surface is exploited, which has random properties and is physically unclonable.

[0149] The last mentioned property, that the surface texture is individual for each object and unclonable, is generated by a random process, can be exploited to the fullest extent for brand protection: for this purpose, on the part of the manufacturer, the texture at predefined points for each manufactured object is detected (e.g. photographed or scanned) and stored, for example, in a database in the cloud.

[0150] To check the authenticity of the object at a later time, the texture of the object is detected again at approximately the same location and a database match is performed. Thus, if there is a hit in the database, i.e. a clear match of the texture, the object is known and verified as authentic. However, if the comparison fails to find a match, the object is not included in the set of parts detected by the manufacturer. In this way, counterfeits can be identified.

Claims

1. A method (20) for identifying whether a predetermined image crop of a first image (10) appears in a second image (12), comprising: The method comprises: Identifying (S21) at least one image feature for each of a plurality of pixels within the predetermined image crop; receiving (S23) the second image (12); Identifying at least one image feature for each of a plurality of pixels of the second image (S24); - comparing (S25) the image features of the second image (12) with the image features of the first image (10) whether both images have the same value; Including, Then, if there is at least one pair of identical image features, it is output that the predetermined image crop is present in the second image (12), otherwise it is output that the predetermined image crop is not present in the second image (12). In the method, The image features are calculated in dependence on pixel values ​​of neighboring pixels; determining a rotationally symmetric surface in the second image and converting the surface into a quadrilateral surface; The image features of the pixels of the second image are calculated from pixels within the transformed surface. A method comprising:

2. After the step of identifying the image features of the first image (10), a step (S22) is carried out of associating with each of the image features of the first image (10) a position of a pixel at which the respective first image feature is identified relative to a reference position of the predetermined image crop, In the comparing step (S25), if the value of the image feature of the first image and the value of the image feature of the second image are identical, the relative position of the associated image feature of the first image is adopted, and the pixel position of the pixel of the second image in which the identical image feature exists is relatively shifted around the relative position and stored; The shifted and stored positions are aggregated pixel by pixel; Then, if a plurality of the aggregated locations are in substantially equivalent pixel locations, an output is made indicating that the predetermined image crop is present in the second image; and Otherwise, an output is made indicating that the predetermined image crop is not present in the second image. The method of claim 1.

3. a plurality of different image crops of the first image are predetermined; said comparing step (S25) being performed for each of said image crops; Overall reliability scale [0010] is identified by the accumulation through the trust. The method according to claim 1 or 2.

4. the plurality of image crops have a predetermined arrangement (11) relative to one another within the first image (10); In the step of comparing, in addition to the aggregated positions, it is further determined what arrangement (13) the aggregated positions have with respect to each other, The two configurations (11, 13) are compared with each other, Outputting whether the image crop appears in the second image is additionally performed depending on whether the two configurations are substantially equivalent. The method according to claim 3.

5. The number and / or location and / or size of the image crops are identified; Initially, a random or predetermined location is selected, The method (20) is performed with respect to a plurality of first images for every image crop using at least one second image; For each pair of the first image and the second image, a confidence level [0025] was identified, the number, position and / or size are varied using an optimization method to increase as much confidence as possible if the first and second images show the same object, and to decrease as much confidence as possible otherwise. The method according to claim 3 or 4.

6. Use of the method according to any one of claims 1 to 5 for checking whether a surface mark of an object belongs to a given object, comprising: the first image and the second image depict a surface of the same object; If at least one image crop appears in the second image, then an indication that the surfaces of the first image and the second image are identical is output. Use of the method.

7. Use of the method according to any one of claims 1 to 5, The method is used to verify the authenticity of a product, the first image and the second image depict a surface of the same product, in particular a surface of the same object, If at least one image crop appears in the second image, then an indication that the surfaces of the first image and the second image are identical is output. Use of the method.

8. Apparatus configured to carry out the method according to any one of claims 1 to 5.

9. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 5.

10. 10. A machine-readable storage medium having stored thereon the computer program of claim 9.

Citation Information

Patent Citations

  • Method for determining the position of a first image area in a correspondence image, SoC and control device and system for carrying out the method, as well as a computer program

    DE102019210580A1

Cited By

  • Cutting management method of medical test paper cutting equipment and related equipment

    CN119550414A