Method and apparatus for determining whether segment of first image appears in second image
By computing the comparison of image features and arrangements, the problems of translation invariance and alignment complexity in markless tracking are solved, and efficient recognition of image segments is achieved, suitable for complex components, and recognition efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202410652465.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art lacks translation invariance in markless tracking, resulting in the recognition result that the query image is consistent with the pose of the reference image, and the alignment steps are complex, making it difficult to apply to rough or lack of visual reference components, and the fingerprint comparison method is calculated. The workload is large and it is difficult to carry out quickly and efficiently in production.
By calculating the unique image features of the image clip, using translation-invariant feature calculation rules, comparing the values and arrangement of image features, using the voting weight matrix to determine the position of the image clip, allowing a certain degree of translation flexibility, suitable for rotary symmetric surfaces, and Compare in the cloud or on-premises to simplify alignment steps.
It realizes translation and unchange recognition of image segments, reduces the computational workload, is suitable for complex or rough parts, improves recognition efficiency and accuracy, is suitable for small image sizes and complex objects, enhances discrimination, and simplifies the production process. .
Smart Images

Figure CN119027762A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method, a device, a computer program and a machine-readable storage medium for determining whether a predetermined image segment of a first image is present in a second image. Background Art
[0002] In production processes, for example in the production of supplier components for the automotive industry, complete traceability of the individual components used in the process is increasingly sought and sometimes even required by law or standards.
[0003] This traceability enables, for example, faster and more precise deductions of the cause in the event of a fault. This makes it possible, for example, to determine the manufacturing machine and the process parameters used. It is also possible to identify or isolate other supplier components involved in the error, and recall activities can be specifically limited to the end product actually involved.
[0004] This traceability in production can actually be ensured by:
[0005] a. Nowadays, DMC is usually used to mark the corresponding individual parts, see https: / / de.wikipedia.org / wiki / DataMatrix-Code, for example by laser engraving, needle embossing, stamping or self-adhesive labeling.
[0006] b. Markerless tracking based on the individual surface texture of individual parts.
[0007] In both cases, a sensor (e.g., a camera) captures a predetermined location of the surface of the part to be tracked, either to automatically read the DMC or to capture a texture. The texture is then compared with a texture previously captured and stored (e.g., in a database with a unique index) at approximately the same location, and for all relevant parts. In other words, during the mentioned database comparison, the current "query" record is compared with a previously captured reference record.
[0008] If the parts are identical, the texture comparison provides high consistency, otherwise very low consistency. Here, for example, 100, 10,000 or 1 million parts are compared. Usually, the comparison must be performed in a rhythm-neutral manner, i.e., it should be performed quickly so as not to slow down the production rhythm, for example 10,000 comparisons in a maximum of 200 ms.
[0009] Markerless tracking has several advantages:
[0010] i. Save effort, cost, investment. For example, laser engraving equipment is much more expensive than a camera including lighting for texture capture, especially when purchased.
[0011] ii. Laser engraving with DMC produces smoke, i.e. small fragments of the surface, which can contaminate components (e.g. nozzles) and thereby impair their functionality. In contrast, markless tracking is completely clean.
[0012] iii. Some components are too small or do not have a suitable surface to apply DMC, and even the smallest 1mm 2 The area is also large enough to capture texture.
[0013] iv. Although DMC can also be read and used by unauthorized malicious persons, this is impossible for textures because only the manufacturer itself knows the allocation of textures. This makes this method more secure.
[0014] v. Prevention of counterfeiting. While counterfeit parts can be equipped with copied DMCs without any problems, it is virtually impossible to physically clone the random surface texture of the component.
[0015] DE 10 2019 210 580 discloses a method for verifying the authenticity of an object, wherein a user photographs an object (e.g. a banknote), for example using a smartphone (smartphone app) and receives a corresponding confirmation from the app shortly thereafter if the object is authentic. The texture comparison is performed, for example, in the cloud, where reference textures are stored in a database and are available for comparison.
[0016] Advantages of the present invention
[0017] The first and most important advantage of the proposed method is its translation invariance. This means that the recognition is invariant (independent) compared to the displacement (translation) of the query object in the camera image. In other words: no matter where the ROI (i.e. segment, Region of Interest) is displaced in the second image, the recognition leads to the same result - of course, provided that the ROI completely contains the counterpart to the first image region. The translation invariance applies to both axis directions (x,y) of the camera image. The necessity to be able to reproduce the pose of the reference ROI in the query image, which has been the case in the prior art so far, is eliminated. The alignment step required by the fragile fingerprinting methods is therefore eliminated in this method.
[0018] The second advantage is scalability, i.e. the user has the possibility to restrict the allowed (x,y) translations. In principle, the method is able to identify an object even if there is only a small overlap between the reference record and the query record with respect to the object. The user can then restrict the translation area (in the sense of the search area) to the desired extent, for example 20 pixels to the left, right, up and down. In this example, the translation vector can therefore lie within an area of 41×41 pixels in size. The advantage of this restriction is the saving of computational effort (computation time), since the computational effort increases with the size of the translation area. However, the computational effort increases much more slowly than in the case of known fingerprinting methods.
[0019] The advantage regarding translation enables the application of the proposed method to object types that have been unmanageable so far using fingerprinting methods, such as rough cast parts or parts lacking available visual references (edges, corners, holes, etc.) that enable a suitable alignment (preprocessing step of uniformly aligning the query image).
[0020] A third advantage of the proposed method is that its similarity metric (described below) is in practice significantly more discriminative than the Hamming distance metric of fingerprinting methods.
[0021] A fourth advantage is that the proposed method works in practice with small image sizes. Even for the most difficult object types, image sizes of, for example, 400x400 pixels are sufficient. In practice, image sizes of around 50x50 pixels are often already sufficient for reliable recognition. Summary of the invention
[0022] In a first aspect, the invention relates to a computer-implemented method for determining whether a predetermined image segment of a first image is present in a second image. The image segment is either manually preselected or determined in another way. The image segment is small, in particular relative to the first image, and particularly preferably the image segment comprises 50x50 pixels up to 400x400 pixels of the first image.
[0023] The method starts with determining image features of a plurality of pixels within a predetermined image segment. The image features are determined according to a predetermined calculation rule, wherein the calculation rule calculates the image features according to the pixel values of the neighboring pixels. For example, the environment of 3x3 neighboring pixels to 127x127 pixels around each pixel can be used for the image features. The calculation rule outputs a unique value, so the image feature is unique, i.e. the image feature characterizes the considered pixel arrangement including its environment. Here, the calculation rule is selected so that even if the pixel value changes at least slightly, for example due to sensor noise or lighting differences, a unique image feature is obtained with a high probability. That is, the calculation rule does not have to determine a unique value for each pixel combination, but should give uniqueness with a high probability. Preferably, the calculation rule is translationally invariant, at least within a small range.
[0024] Subsequently, a second image is acquired, in particular a segment (ROI) of the second image which is smaller than the second image but at least as large as the image segment of the first image.
[0025] Then, the image features of the second image or the plurality of pixels in the ROI are determined. The image features of the first image and the second image are calculated in the same way, that is, the same calculation rules are used for the image features.
[0026] The determined image features of the second image are then compared with the determined image features of the first image, preferably taking into account not only the values of the image features but also the relative arrangement of the image features in the respective images. If a group of at least two different image features is found in the second image, which have an identical or almost identical arrangement to a group of different image features of the same value in the first image, then it is output that the predetermined image segment is present in the second image, otherwise it is output that the predetermined image segment is not present in the second image.
[0027] The particularity of the method is that a rotationally symmetric surface, in particular a circular or circular surface, is determined from the second image and in particular from the first image, and the surface is transformed into a quadrilateral surface, in particular a rectangular surface, and image features of pixels of the second image and in particular the first image within the transformed surface are calculated. The rotationally symmetric / circular surface can be transformed into its polar coordinate version with a predetermined size - height x width - by a given center point and a radius "start" and a radius "end" and preferably a start / end angle. It should therefore be noted that the predetermined image segment comes from the transformed surface of the first image.
[0028] It is conceivable that when calculating the image features of these original adjacent pixels, the adjacent pixels of the rotationally symmetric surface that are now separated from each other in the quadrilateral surface and located at the opposite ends of the quadrilateral surface are incorporated into the calculation rules. In other words, it can be provided that the quadrilateral surface is supplemented at its ends with copies of the pixels at the opposite ends, for example.
[0029] It is proposed that after the step of determining the image features of the first image, an assignment step is performed, which assigns to each image feature the relative position of the following pixel with respect to a reference position of a predetermined image segment, for which the corresponding first image feature has been determined. The reference position may be the center point of the image segment. In the comparison step, if the values of the image features of the first image and the second image are the same, the relative positions assigned to the image features of the first image are adopted and the pixel positions of the pixels of the second image with the same image features are stored in the voting weight matrix in a manner relatively shifted by the relative positions (particularly at the shifted pixel positions). Then, the shifted stored positions are aggregated pixel by pixel, wherein if the vast majority of the output positions exist at substantially the same pixel position of the second image with respect to the confidence measure, then the output predetermined image segment exists in the second image, otherwise the output predetermined image segment does not exist in the second image.
[0030] Essentially the same pixel position may be understood as a deviation of + / - 2, + / - 3 or + / - 4 pixels or more.
[0031] Optionally, an output step may then be performed in which the pixel position having the largest count after aggregation is output as the position of the image segment in the second image.
[0032] Optionally, to make evaluation easier, the aggregate can be smoothed, whereby a clear maximum is achieved at a unique location while unimportant secondary maxima are suppressed.
[0033] Preferably, the image feature is used to address a lookup table, wherein the assigned relative position is stored for the respective address. Optionally, in addition to the relative position, the lookup table also contains weights, wherein the weights change, in particular decrease, as the distance between the position of the image feature and the reference position increases. Additionally, the weights can also be used during aggregation to aggregate in a weighted manner.
[0034] In the case of multiple image segments, each of the above method steps is performed for each different image segment. Preferably, the image segments are evenly distributed on the object texture in the first image, for example, evenly distributed on a circle.
[0035] Furthermore, it is proposed that a plurality of image segments have a predetermined arrangement relative to one another within the first image, wherein in the comparison step, in addition to the stored positions, it is also determined how these positions are arranged relative to one another, and the two arrangements are compared with one another. The arrangement can be given by defined distances between the positions or between the topological patterns. If the topological patterns are compared, then preferably only the structure of the patterns is compared and the orientation of the patterns is not taken into account. The advantage in this case is that the decision whether the objects imaged onto the images are identical is effectively made more reliable by the further independent decision criterion of the topological arrangement.
[0036] It should be noted that the step of determining the image features of the pixels of the second image is performed immediately after the second image has been captured, preferably in a computing unit, and transmitted to a server which sends back a signal based on the comparison. Preferably, the image features of the image segments are already determined in the cloud or have been transmitted to the cloud and are only compared in the cloud with the received image features. As an alternative to the cloud, the method can be performed locally.
[0037] In the first aspect of the present invention, the first image and the second image preferably image the surface of the same workpiece.
[0038] The first image and the second image can be recorded with the same camera or with the same multiple cameras. No special requirements need to be made for the cameras that capture the two images, a low image resolution is sufficient. In both cases, the recording arrangement should be designed to be as similar as possible in terms of the choice of lens, the distance from the selected object surface, the focus on the object surface, the type and arrangement of the lighting device. The exposure time and the lighting intensity should be selected so that the corresponding image has no noticeable motion blur and is neither too dark nor overly controlled. If it is not possible to use two identical camera arrangements, the differences can be compensated for afterwards by image processing. For example, if the second camera has a different distance or a different focal length or a different pixel format (Pixel Pitch) from the first camera, the resulting scaling differences can be compensated by image scaling so that the object surfaces considered subsequently all exist at approximately the same size (in pixels).
[0039] In a further aspect, the invention relates to the use of one of the methods of the first aspect for checking whether a surface record of an object belongs to a pre-given object. Preferably, a query object is compared with all reference objects. The best consistency (highest overall confidence value) is then determined. If this value is high enough (above a threshold), a match is output. However, if the best overall confidence value is not high enough, no match is found. The query object is either fake or has not yet been captured as a reference.
[0040] In a further aspect, the invention relates to the use of one of the methods of the first aspect for verifying the authenticity of a product.
[0041] In further aspects, the invention relates to a device and a computer program, both of which are arranged to perform the above method, and to a machine-readable storage medium on which the computer program is stored. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The embodiments of the present invention are explained in more detail below with reference to the accompanying drawings. In the accompanying drawings:
[0043] Figure 1 Schematically showing images of the produced components, each image having a plurality of image areas AL arranged in a grid;
[0044] Figure 2 An embodiment of a method for re-identifying an image region is schematically shown. DETAILED DESCRIPTION
[0045] There are already various methods for markerless tracking based on surface texture, such as the so-called Tracking and Tracking Fingerprinting from the Fraunhofer Institute for Physical Measurement Technology. This technology is based on the fact that many semi-finished products or components have a microscopically unique surface structure or color texture. A defined area of the component is selected and recorded with high resolution using an industrial camera. From the image recording with specific structures and their positions, a digital identifier is calculated and an ID is assigned to this digital identifier. This pairing is stored in a database together with other data such as measurement data or creation data. For later identification, the entire process is repeated and the data comparison returns the ID and other individual features of the component.
[0046] The "computed digital identifier" referred to is hereinafter referred to as a fingerprint. A fingerprint can be imagined as a number having many digits, or equivalently as a long sequence of bits.
[0047] During the mentioned database comparison, the current fingerprint is compared with previously captured reference fingerprints and the degree of difference is determined in each case. Ideally, exactly one reference fingerprint is found here that differs slightly from the query fingerprint, while the comparison with all other reference fingerprints provides a large difference. Exactly in this way the identification is considered successful.
[0048] Typically, the difference between the physical reference object and the physical query object is determined solely based on the fingerprint pairing, for example as the Hamming distance of two bit sequences or defined according to another distance metric.
[0049] In this regard, the difference between the present invention and the prior art is particularly obvious.
[0050] Methods known from the prior art for traceability and brand protection are based on the formation of so-called fingerprints, which describe individual texture properties of an ROI (region of interest) of an object, for example in the form of a bit string of a fixed length, which may be, for example, several thousand bits.
[0051] Then the texture comparison is performed as the fingerprint comparison. For example, the Hamming distance between the fingerprint of the reference object and the fingerprint of the query object is formed, that is, the bit-by-bit comparison is performed and the differences are counted. The smaller the Hamming distance (ideally 0), the more similar the objects are.
[0052] If the Hamming distance is normalized by the length of the bit string, one obtains similarity values between 0 and 1, where in practice these values only exploit the range of 0 to about 0.5, since if the bit strings are statistically independent, only about half of the bits will differ.
[0053] Comparison of fingerprints, such as by Hamming distance, has the advantage of being fast, so many comparisons can be performed in a short time.
[0054] However, it also has a significant disadvantage, because the result of fingerprint comparison is not translation invariant. For example, if the fingerprint of the query object is not formed at the same part (ROI) as the reference object, but is offset by, for example, 20 pixels, then the fingerprint no longer has any similarity to the reference fingerprint.
[0055] Therefore, texture comparison in this case provides erroneous results: misclassifying texture image pairs as dissimilar despite involving the same object.
[0056] In practice, this unfavorable behavior caused by the lack of translation invariance of fingerprinting methods leads to the fact that a lot of additional work has to be regularly done to make the pose of the ROI in the reference and query images as identical as possible, since deviations of just a few pixels are no longer tolerated.
[0057] For some object types, due to the geometry of the component, the object can be mechanically placed in a reproducible pose relative to the camera. This generates additional work (robotic gripper grasps the object, places the object in a matching template in front of the camera, re-grasps the object and puts it back, time requirements, costs, sensitivity). For other object types, these geometric conditions are not given and visually salient features of the object type must be exploited to make the ROI reproducible, such as the pose of holes, grooves, corners, edges, etc. This also generates additional work (image analysis algorithm adapted to the component, applied by experts).
[0058] However, there are also some object types for which neither one method nor the other can reproducibly find the ROI.
[0059] This applies, for example, to objects of a rougher nature, such as metal cast parts, which have no defined edges but rather strongly beveled and rounded edges as a result of being removed from the casting mold and which may have more complex, random casting burrs.
[0060] Even for planar object types such as metal sheets, paper, cardboard, fiber plastic sheets, etc., reproducibility of the ROI pose may be difficult or even impossible, especially if during processing the edges are bent, folded, punched or cut, whereby a possible visual reference is lost.
[0061] Small uncertainties in the ROI pose can be addressed by forming multiple fingerprints with slightly offset ROI positions (e.g., pixel-by-pixel shifts). However, this will correspondingly increase the amount of work required for fingerprint formation and fingerprint comparison exponentially, often even quadratically, since there are uncertainties in both axes (x, y). Large uncertainties about the ROI pose cannot actually be covered.
[0062] In order for the fingerprint method to work in operation, two steps must always succeed, namely the alignment step (also called the matching or alignment step) and the fingerprint comparison step.
[0063] What is particularly difficult is that the alignment step that must be performed at each camera station (the first station for first capture and a certain number of other stations for identification) for ROI alignment must always produce the same results. This is difficult to guarantee in practice, especially during continuous operation.
[0064] Fingerprint methods are therefore considered fragile in practice and are accordingly unpopular with users. For example, a slight displacement of the camera relative to the stored template can result in the component no longer being identifiable at the recognition station, and production is therefore at a standstill.
[0065] In the following, the following confidence measure shall be introduced, which evaluates the trust in the quality of the found correspondences.
[0066] Confidence measure q B Mathematically it is defined as: Where p1 is the height of the best peak and p2 is the height of the second best peak. Here, peak refers to the value of the global or local maximum in the voting weight matrix or in a further processed form of the voting weight matrix.
[0067] A preferred further processing of the voting weight matrix consists in smoothing the voting weight matrix after the accumulation (collection of voting weights) has been completed. A suitable smoothing filter is, for example, a two-dimensional Gaussian smoothing filter.
[0068] Since p1 ≥ p2 is always satisfied, q B ≥0.
[0069] Confidence measure q B It is suitable, for example, to evaluate whether the corresponding object search is successful already from a single image comparison (one-to-one comparison: 1 query image query ROI vs. 1 reference image region). If a corresponding object is found, then q B Larger (e.g. q B =11), otherwise q B Smaller (e.g. q B =0.1).
[0070] In an alternative implementation of the confidence metric, the subtraction by 1 may be eliminated.
[0071] An alternative simpler confidence measure q A Yes: q A = c·p1, with an optional constant c. Thus, the height of the best peak is used as a confidence measure.
[0072] This again means the global maximum in the voting weight matrix or in a further processed form of the voting weight matrix (e.g. after smoothing filtering). An optional constant c can be used to set q A The value range of A ≤1.
[0073] Confidence measure q A The advantage is that the determination of the second peak value p2 can be omitted, which means a saving in arithmetic operations. On the other hand, the comparison value for normalization is also eliminated thereby.
[0074] However, this apparent disadvantage becomes insignificant in 1-to-N comparisons (1 query image to many (N) reference images, considered in more detail later), since here there is typically one hit versus N-1 non-hits, with hits being represented by high q A The value is characteristic, and all non-hits have a lower q A This information is sufficient to determine the winner, so the height of each second-best peak in the same query image represents discardable information.
[0075] Therefore, a simpler confidence measure q A Below is the preferred confidence metric.
[0076] In order for the invention to function reliably, the camera is preferably used to capture a two-dimensional image in the form of grayscale values or multidimensional values (color values) of the surface (parts) of the object (so-called texture capture) and this image is as reproducible as possible for different cameras. In other words, the image should look as similar as possible. Easily correctable differences do not cause problems here, which can be corrected, for example, by rotation, scaling, shifting or brightness adaptation of the image content.
[0077] In order for the image contents of two recordings of the same object to be as similar as possible in this sense, the two recording arrangements should be as similar as possible or at least coordinated with one another. This relates in particular to the lighting.
[0078] For objects with matte surfaces, this is relatively easy to achieve, since the incident light is already scattered, so that the imaging of the object hardly changes with slight changes in the lighting position. For objects with strongly reflective surfaces, such as polished or brushed and then chrome-plated steel, the imaging depends strongly on the position of the light source, especially if the light source is point-shaped. In order for the two recording arrangements to still provide reproducibly similar imaging of the object, it is strongly recommended to choose an illumination in which the light hits the surface as evenly as possible from all directions (diffuse illumination). This is achieved, for example, by dome lighting close above the object, in which the interior of a matte white hemisphere is illuminated, resulting in an indirect illumination that is as non-directional as possible. Here, the camera is located inside the hemisphere or observes the interior of the hemisphere through a hole in the region of the symmetry axis of the hemisphere.
[0079] Especially for difficult surface types of this kind, it may make sense to calculate the texture image from multiple consecutive recordings with the same camera, each under different illumination. For example, four recordings can be made in rapid succession, with only one quadrant of the annular or dome-shaped diffuse illumination being switched on each time. Suitable known methods are, for example, "Shape from Shading" or "Photometric Stereo".
[0080] For example, in the case of shape from shading, a so-called texture image describing the reflectivity of the surface and a curvature image describing the local curvature can be calculated. Both images are suitable for the present invention independently of each other or in combination, since they each provide a reproducible imaging of the object for which the dependence on the lighting has been eliminated.
[0081] In order to find an image region in a second image from a first image, a number of methods are known, for example from DE 10 2019 210 580.
[0082] One of these methods first defines a small outline of the image segment, such as a square outline or a circular outline or a polygonal outline.
[0083] The feature points are located within this contour: these are image features formed by their respective environment, each image feature being represented, for example, with 16 bits. Such image features are therefore a compact representation of a more or less local image content. Overall, therefore, the larger overall image area contributes to the totality of the formed features. To compute the image features for a given pixel, latch descriptors can be used: G. Levi and T. Hassner, LATCH: Learn arrangements of three patch codes, IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-9, 2016. However, it should be noted that there are a variety of other known methods for computing these features.
[0084] In this embodiment, the size of the square is 50×50 pixels and contains 2500 feature points. In other words, the feature density here is 1 feature per pixel. However, the feature density can also be higher or lower, such as 4 features per pixel or 1 feature per 9 pixels.
[0085] The corresponding feature value (eg, with a word length of 16 bits) is used as an address in a lookup table, into which the feature position in the image is then entered relative to a reference position in the image, optionally together with other information, such as weight.
[0086] The processing of the considered image region from the first image is thus completed when at the latest all 2500 feature points have been entered into the table (a smaller number is sufficient in most cases).
[0087] The second image is an image in which a position corresponding to a reference position should be determined. The search area for the position may include the entire second image. Features are then generated for the entire second image accordingly. Alternatively, the search area includes only a portion (ROI) of the second image. Then, for the second image, features are generated for the surface corresponding to the enlarged ROI, wherein the surface is enlarged by the allowed displacement, for example, 20 pixels to the left, right, upward, and downward, respectively. For the second image, either the feature calculation method exactly the same as for the first image is applied, or an adapted method that appropriately takes into account changes in texture state (for example, from dry and clean to wet with oil) is applied.
[0088] Here, prior knowledge can be used which, for example, describes the change in texture in the image during the transition from dry to oil-wet or vice versa, for example statistically, for example in the form of a complex distribution density of local contrast (or color values) between the two states.
[0089] In this embodiment, the feature density in the second image is also 1 feature per pixel. The feature density can also be higher or lower and does not have to be consistent with the feature density of the first image.
[0090] The features generated from the second image are again used to address the lookup table, but this time in a read mode. As described above, starting from the current feature position in the second image, voting weights are given at relative positions determined based on the lookup table and entered into the voting weight image. This can be performed for each feature in the search area, where the votes are collected (accumulated) in the voting weight image, preferably summed or weighted.
[0091] In a preferred embodiment, the size of the voting weight image roughly corresponds to the size of the search area. The voting weight image can also be chosen to be smaller or larger. However, it does not have to be chosen to be larger than the range of relative positions when voting. This range depends on the size of the first image area and the relative pose of the reference position - or in other words: on the length limit of the 2D vector.
[0092] In the case of the method for retrieving an image region, in the case of the presence of a counterpart in the voted weight image, a very distinct and locally very concentrated accumulation point is generated, the pose of which corresponds to the sought position, i.e. in the second image to the reference position from the first image. By evaluating the accumulation point, for example in the sense of determining the position of its maximum or the position of its center of gravity, the position of the sought counterpart can be determined with sub-pixel accuracy, if this is desired or necessary. Depending on the resolution of the camera, this can correspond to sub-millimeter accuracy.
[0093] The determined position information can be used advantageously in different ways: From the difference between the reference position in the first image and the position found in the second image, it can be determined how the pose between camera and object differs between the two camera stations. Statistics can be determined from this. The mean value provides information about differences between different cameras that one might want to eliminate easily. Deviations from the mean value (e.g. expressed as standard deviation) provide information about how the pose of the individual objects fluctuates. These position fluctuations can be caused by variations in the gripping of the robot or by differences in the shape of the component. For example, for undeburred cast components, such variations can be large.
[0094] Statistics also help parameterize the search area, since, as mentioned above, it may make sense not to fully exploit the degrees of freedom of translation but to configure a smaller search area to reduce the computational effort. However, the search area should not be chosen so small that it misses the counterparts.
[0095] If multiple counterparts are searched per image pairing, conclusions can be drawn based on the arrangement of the found counterpart positions and the confidence of the classification can be further improved.
[0096] Next, consider how advantages may be gained from the optional use of multiple reference image regions.
[0097] A number of reference image regions are placed in the reference image. For example, M=12 reference image regions are placed relative to the reference image in such a way that they preferably capture parts of the object surface that enable differentiation. Noise textures that arise from random processes are particularly suitable for identifying individual objects. In contrast, regions that appear almost identical on every object are unsuitable. Surfaces that are not suitable for positioning reference image regions are, for example, overly controlled image regions that are so bright that the camera can no longer distinguish details, or very dark image regions in which the signal-to-noise ratio is so low that the image signal is mainly determined by the sensor noise (which of course does not contribute to the recognition in any way).
[0098] Unsuitable surfaces can also be, for example, nearly homogeneous image areas where the signal is approximately constant or only slightly varies locally, i.e. no or only little information helps with recognition, or very blurry captured image areas that are clearly outside the focal plane of the camera, or image areas where every object looks the same and therefore cannot be used for differentiation, such as edges, corners, holes. This also includes shadow edges caused by lighting.
[0099] Also unsuitable are surfaces that are damaged during the process of tracking the object or that are so severely altered that they cannot be tracked, such as surfaces that have been milled, ground or sandblasted, or surfaces that have been coated with opaque varnish and otherwise obscured so that they are no longer visible afterwards.
[0100] The reference image area is usually set once for a component type after the recording arrangement (camera, lighting, pose, image resolution, image segments) has been determined. This setting is usually the only parameterization step still required. The setting of the reference image area then normally applies to all objects of this component type. This setting can be done by an expert or automatically.
[0101] For example, an expert may set and use M=12 reference image areas, whose centers are placed on a circle at equiangular intervals of 30°, like the complete hours on the face of a clock. This arrangement is suitable, for example, for circular or annular object surfaces. The reference image areas are preferably located within a well-focused, randomly textured surface. Edges and blurred image areas are avoided.
[0102] Overlap of reference image regions, while permitted, should be avoided as the additional work does not provide additional benefit.
[0103] In the simplest case, the sum or average of the M confidence measures is formed as the overall confidence measure q when multiple image regions are used Ges ,For example:
[0104]
[0105] The individual confidence measure q i Here it is preferred to be able to A Or according to q B or formed according to other rules.
[0106] The advantage of this method is that a plurality of small reference image regions can be placed independently on suitable sub-surfaces, each of which has a suitable surface texture, thereby achieving an optimal adaptation to the component type in a distinguishable sense with little effort.
[0107] Using an expert to set the reference image region may be disadvantageous due to cost or availability reasons.
[0108] As an alternative, it is possible to choose to automatically and optimally set the image regions. For this purpose, training with a very small number of objects is required, for example N R = 10, but at least N R =2.
[0109] Let's assume that N R For each of the N training objects there exists a reference record (hence the index R) and these N R N of the training objects Q ≤N R There is a query record (index Q) for each.
[0110] Here 1≤N is satisfied Q ≤N R , or in other words: ideally there is also a query record for each object, that is, N Q =N R , however, if necessary, a single query record is sufficient, i.e. N Q =1.
[0111] For example, training is as follows:
[0112] 1. Initially place a predetermined number M of reference image regions, eg randomly or regularly (eg as tiles).
[0113] 2. Apply this placement to all N R Then the application is used to convert N Q query image and N RHere, N reference images are generated. R ×N Q Similarity values are entered into a N-th dimension. R ×N Q The similarity matrix is , where the column index corresponds to the query number and the row index corresponds to the reference number.
[0114] In N R =N Q And the order is the same, the correct assignment will be on the main diagonal.
[0115] 3. Evaluate the similarity matrix. The goal is to optimize the similarity matrix and achieve the maximum value in each column (usually on the main diagonal) in the case of a correct assignment, while all other values in the column are as small as possible. For example, the measure can be the quotient or difference of the highest similarity and the second highest similarity (in the corresponding column), averaged over all columns. This results in a total number of evaluations for the placement settings, which is suitable for maximization.
[0116] 4. If the total number of evaluations has reached the maximum value or exceeded the threshold, the optimization is complete and the determined parameters are thus optimized. Otherwise, continue.
[0117] 5. The setting is constantly changing. This includes changing the following: the position of the individual reference image regions (i.e. their position in the image), and / or the size of the individual reference image regions (e.g. in the case of rectangles or ellipses: width, height, aspect ratio, angle), and / or the number of reference image regions.
[0118] It should be noted that here again overlapping of reference image regions is avoided as much as possible.
[0119] Of course, the usual methods used in optimization can be used to speed up this training or bring it to a global optimum, for example, determining the gradient to determine and, when changing, prioritize the parameters that have the greatest impact on the total number of evaluations.
[0120] It is advantageous to also include the computational effort in the overall evaluation number (less computational effort leads to better evaluation numbers). This therefore automatically leads to the number and size of the reference image regions not becoming larger than necessary and that they do not overlap.
[0121] Objects are usually designed to be essentially rotationally symmetrical. It is therefore usually not always possible to reproducibly position the object in front of the camera at the same angle (around the axis of rotational symmetry). For example, although side notches can provide clarity and thus provide assistance, this aid is not available for some object types.
[0122] For such objects and similar objects in this sense which are ambiguous about rotation about an axis, the present invention has a strong advantage over the prior art, since their translational invariance in two directions can be converted into tangential invariance and axial invariance. In this case, the tangential invariance, which can also be called rotational invariance, is particularly important.
[0123] For this purpose, it is recommended to digitally unfold a segment of the image, for example a circular surface. To do this, the two radii of the disk and the position of its center point in the image must be set. The center point should at least approximately coincide with the point of penetration of the axis of rotational symmetry in the image.
[0124] The parameters for cutting the disk are either known in advance or can be determined from the image. In particular, if the center point of each recording changes, for example based on an estimate of the center of symmetry, it may be meaningful to redetermine the center point of each recording. In this case, an approximate estimate (for example accurate to a few pixels) is sufficient.
[0125] When the disk is digitally unrolled, the disk is cut and rolled into a rectangular shape. Therefore, the pixel values (grayscale values, color values) are correspondingly distorted.
[0126] It should be noted that, when unfolding, not only a surface such as a circular arc can be unfolded, but of course the surface to be unfolded can also be an inner / outer cylinder or an inner cone (truncated head) or an outer cone (truncated head). Even shapes such as barrels, dumbbells or waves can be imagined. Thus, the unfolding can generally be applied to rotationally symmetric surfaces.
[0127] At the cutting line, it can be imagined that the rectangle continues cyclically on the left and right sides, respectively, because these two ends belong together. Accordingly, all image processing operations and further processing steps beyond this cutting line can continue uninterrupted.
[0128] After unfolding the disk-shaped image content into a rectangular image, the method can be applied to the rectangular image as usual: namely, comparing the unfolded query image with the unfolded reference image, respectively.
[0129] If the described principle of cyclic continuation is always followed, the result of the detection according to the invention is independent of the position at which the radial cut is made on the ring.
[0130] Assuming the circular component is in an unknown angular position, the unknown angular position becomes an unknown horizontal translation by unfolding. However, the algorithm is invariant to translation. This means that the result is independent of the horizontal translation. Therefore, despite the unknown angular position, the object can still be recognized in its unfolded form. Therefore, the translation-invariant method becomes a rotation-invariant method by unfolding.
[0131] In the expanded image there is also vertical translation invariance. This means that vertically displaced counterparts can also be found. The vertical translation invariance is mapped here to an invariance in the radial direction. This invariance is also very useful, because it can compensate for an inaccurately placed center point. In other words, if the estimate of the symmetry center is wrong, for example misplaced by 5 pixels, and the expansion of the disk is correspondingly carried out around the wrong center point, this is not a problem, because the (thus vertically displaced) counterpart can still be found. This makes the robustness of the method significantly increased further.
[0132] In summary, in other words, for the recognition of essentially rotationally symmetric objects in practice this means that the angular position of these objects when the images are recorded can be arbitrary and unknown and an approximate knowledge or estimation of the position of the rotation axis or center point is sufficient.
[0133] This important advantage over the prior art is achieved by the two-dimensional translation invariance given by the method, which can be converted into rotation invariance and invariance in the radial direction by unfolding.
[0134] The invariance in the radial direction is not really invariant in the mathematical sense, since the expansion is a nonlinear mapping with a scaling factor that depends on the corresponding radius. Errors in the pose of the center point translate into scaling errors accordingly. However, in practice this apparent problem has been shown to be negligible and therefore insignificant.
[0135] Once a plurality of reference image regions are used for each image, an optional additional possibility results for further increasing the confidence level of the recognition.
[0136] This is because the M reference image regions are in a particular arrangement in the reference image. It can then be expected that the correct counterparts that can be found are in the same or similar arrangement in the query image. This can be checked to thereby form an additional or alternative quality metric.
[0137] If the arrangement is as expected, the confidence level is high, and deviations from expectations correspondingly reduce the confidence level.
[0138] However, if not all M possible correspondences can be found, for example because the object is locally damaged, at least the remaining parts of the correspondences should still be in the expected sub-arrangement. This should then still result in a relatively high confidence.
[0139] This is Figure 1. M = 12 image regions are set on a reference object 10, here marked with the letters AL, which are to be used for texture-based recognition. The center points of the image regions are in a predefined arrangement 11, which is also shown as a graphic on the right side next to the image. In this case, it is a grid-like arrangement of rectangular grids with a constant grid spacing (in both dimensions). In the possible 4×4 grid space here, 4 corner positions are not used.
[0140] Other arrangements of the image regions are conceivable, for example on a circle or other geometric shapes.
[0141] At the moment of consulting the recording, the same object 12 now has apparent damage, thereby making identification difficult. Other factors, such as changes in recording conditions, may make things more difficult.
[0142] All this results here in that some counterparts, such as G, are not found at all, and some counterparts, such as F, have very low or strongly reduced confidence, such as B.
[0143] Especially for such difficult cases (which may be more difficult than those shown here), it is advisable to introduce additional checks, in which counterparts of the arrangement are checked in order to derive additional evaluation metrics therefrom.
[0144] This should be explained with this illustrative example. The correspondences E and I found are considered particularly strong here. For example, the positions of these correspondences can then be used as anchor points to correspondingly anchor a known arrangement, in this case a grid, as shown in arrangement 13 on the right. This results in a slight rotation of the grid to the right (because the object is correspondingly rotated in the query record). It may also be necessary to slightly scale (reduce or enlarge) the grid here.
[0145] Starting from a grid anchored in this way, all remaining expected counterpart positions, such as the position of B, can be determined. If the counterpart is actually found within a circle around the expected position, this should have a positive effect on the confidence of the combination, i.e. increase it. For example, the circle takes into account expected measurement inaccuracies, image distortions, numerical inaccuracies and error propagation. The circle can have a fixed or variable radius. Optionally, the circle grows with increasing distance to the anchor point, which can also be reproduced in arrangement 13, where the circles have different diameters. Optionally, the closer the anchor points are to each other, the more the circle grows (leverage effect is reduced, not shown graphically here).
[0146] The circles are not to be understood literally, so they do not have to be circular areas, but can also have other shapes. In particular, elliptical shapes are relevant, since the elliptical shape is a result of the error calculation.
[0147] Then for example the confidence of the combination can be calculated as follows:
[0148] 1. Select two found counterparts, for example E and I, to fix the grid (here synonymous with the expected arrangement) in its position, and by appropriately shifting, scaling and rotating the grid.
[0149] 2. Check all other found counterparts AD, FH, JL to determine whether they fit into the anchored grid, e.g. whether they lie within their respective expected circles.
[0150] 3. In the simplest case, what matters is that the expectations are met. Here, count values from 0 to M-2=10 may be generated as new confidence measures, where the anchor points themselves are not counted.
[0151] In an improved variant, the count is replaced by the sum of M individual confidence values, each of which is calculated based on q A or q B The corresponding objects are determined and are located within their respective circles. Optionally, the individual confidence values of the anchor points can also be added.
[0152] The formula for this could be, for example:
[0153]
[0154] where if the i-th counterpart lies within the expected circle or it is an anchor point, then w i =1, otherwise w i =0.
[0155] In a further improved variant, the circle criterion is to be understood more gently, i.e. not as a yes / no decision (inside the circle or not), but rather as an evaluation of the distance of the found corresponding counterpart position to the corresponding expected position, e.g. using the maximum value for a direct hit, a medium value for a slightly greater distance, and approximately 0 for an excessive distance.
[0156] q Ges The formula for also applies here, but instead of using binary-valued weighting factors, a scalar-valued weighting factor w is used i , the weighting factor evaluates the pose of the i-th counterpart.
[0157] For example, a sampling of a two-dimensional correlated or uncorrelated Gaussian density or a two-dimensional conical density function may be used as w i The two-dimensional correlated or uncorrelated Gaussian density and the two-dimensional conical density function have their maximum values at the expected positions, respectively. The evaluation number w obtained by scanning the density function at the position of the found counterpart is ican be multiplied by the corresponding individual confidence value q i (eg peak height). The M-2=10 products so far are added again. Optionally, the anchor points can also be included in the sum again, preferably with a direct hit evaluation in each case.
[0158] Optionally, the last mentioned step can be repeated with all other possible anchor points, for example next with anchor points C and L. The further apart the two anchor points are, the more leverage there is for anchoring the grid, which is advantageous.
[0159] These permutations avoid the risk of an unreasonably poor overall assessment due to false anchoring caused by the wrong counterpart. Ges Among them, it is preferred to further use the overall confidence with the highest achieved value, because for this overall confidence it is best to assume that the underlying anchor is correct.
[0160] For the embodiment considered here (with M=12 image regions), the complete set of possible arrangements can be considered without any problems. If only a subset of arrangements is to be considered, it is advantageous to preferentially select anchor points with high individual confidence values and at the same time long levers.
[0161] So far, only the case with two anchor points has been considered, which already enables the adaptation of rotation and scaling between the reference image and the query image. However, it is also possible to work with a different number of anchor points, for example three or four. Thus, for example, even projections onto the object surface with different camera perspectives can be allowed and taken into account. This provides the user with more freedom when installing the facility. For example, the user then no longer needs to take care that the viewing direction of the camera at the respective station is oriented perpendicular to the object surface, but an inclined viewing angle and even hand-held camera operation are also possible.
[0162] Figure 2 A flow chart of a method for identifying image regions is shown.
[0163] The method begins by determining ( S21 ) image features of a plurality of pixels within a predetermined image segment.
[0164] Optionally, an assignment S22 is additionally performed. In this case, each image feature is assigned the relative position of the pixel for which the corresponding first image feature was determined, relative to a reference position of a predetermined image segment.
[0165] Subsequently a second image is obtained (S23), wherein a rotationally symmetric surface is determined from the second image and transformed into a rectangular surface, and image features of a plurality of pixels of the second image are determined (S24). As described above, the first image is also transformed.
[0166] The determined image features of the second image are then compared with the determined image features of the first image (S25) to determine whether the two have the same value. Then, if there is at least one pair of identical image features, it is output that the predetermined image segment exists in the second image, otherwise it is output that the predetermined image segment does not exist in the second image. It should be noted that, for example, for image features that are only conditionally unique, it is preferred that there should be at least 2 or more pairs of identical image features. Preferably, multiple pairs of identical image features exist at the same position in the voting weight matrix.
[0167] If step S24 has been performed, in the comparison step (S25), if the image feature values of the first image and the second image are the same, the relative positions assigned to the image features of the first image are adopted and the pixel positions of the pixels of the second image where the same image features exist are stored in a manner relatively shifted by the relative positions. As described above, these specific positions can be regarded as voting weights, which are added pixel by pixel in a voting weight matrix (voting image). The position of the image segment in the second image can be determined with the help of the voting weight matrix.
[0168] according to Figure 2 The two most important areas of application of the method are:
[0169] Mark-free tracking (Traceability) based on the individual surface texture of components in industrial production, for example. Brand protection is achieved by using the individual surface texture of the product to identify genuine products of own production and distinguish / detect third-party products or counterfeit products.
[0170] In both cases, exploitation is made of the naturally occurring texture on the surface of the object, which has a random character and is virtually physically impossible to clone.
[0171] The last-mentioned property, namely the creation of an individual surface texture for each object that cannot be cloned by a random process, can be used for brand protection. To this end, the manufacturer captures (e.g. photographs or scans) the texture at predefined locations for each object produced and stores it, e.g. in a database in the cloud.
[0172] To later check the authenticity of an object, the texture of the object is recaptured at approximately the same location and a database comparison is performed. Thus, if there is a hit in the database, i.e. the texture is clearly consistent, the object is known and verified as authentic. However, if no consistency is found in the comparison, the object does not belong to the set of parts captured by the manufacturer. Counterfeits can thus be identified.
Claims
1. A method (20) for determining whether a predetermined image segment of a first image (10) is present in a second image (12), comprising the following steps: Determining (S21) at least one image feature for each of a plurality of pixels within the predetermined image segment; Acquiring (S23) the second image (12); Determining (S24) at least one image feature for each of a plurality of pixels of the second image; comparing (S25) the image feature of the second image (12) with the image feature of the first image (10) to determine whether the two have the same value, If at least one pair of identical image features exists, it is output that the predetermined image segment exists in the second image (12), otherwise it is output that the predetermined image segment does not exist in the second image (12), It is characterized in that Calculating the image features based on pixel values of neighboring pixels, and determining a rotationally symmetric surface in the second image and transforming the surface into a quadrilateral surface, and Image features of pixels of the second image are calculated from pixels within the transformed surface.
2. The method according to claim 1, characterized in that After the step of determining the image features of the first image (10), an assignment step (S22) is performed, which assigns to each image feature of the first image (10) the relative position of the pixel for which the corresponding first image feature has been determined, with respect to a reference position of the predetermined image segment, In the comparison (S25) step, if the values of the image features of the first image and the second image are the same, the relative positions assigned to the image features of the first image are adopted and the pixel positions of the pixels of the second image having the same image features are stored in a manner relatively shifted by the relative positions, wherein the shifted stored positions are aggregated pixel by pixel, wherein if a plurality of aggregated positions exist at substantially the same pixel position, then it is output that the predetermined image segment exists in the second image, otherwise it is output that the predetermined image segment does not exist in the second image.
3. The method according to claim 1 or 2, characterized in that: A plurality of different image sections of the first image are predefined, wherein a comparison step (S25) is performed for each image section, wherein an overall confidence measure is determined by accumulating the confidences 4. The method according to claim 3, characterized in that A plurality of image segments have a predetermined arrangement (11) relative to one another in the first image (10), wherein in a comparison step it is determined in addition to the aggregated positions which arrangement (13) the aggregated positions have relative to one another, and wherein the two arrangements (11, 13) are compared with one another, wherein the output of whether the image segment appears in the second image is additionally dependent on whether the two arrangements are substantially identical.
5. The method according to any one of the preceding claims 3 or 4, characterized in that The number and / or position and / or size of the image segments are determined, wherein a random position or a predetermined position is initially selected, wherein the method (20) is performed for all image segments of a plurality of first images and with at least one second image, wherein a confidence value is determined for each pairing of a first image and a second image The number, position and / or size are changed by means of an optimization method so that when the first image and the second image show the same object, the confidence factor is as large as possible, and otherwise the confidence factor is as small as possible.
6. A method according to any one of the preceding claims for checking whether a surface record of an object belongs to a predetermined purpose of the object, wherein a first image and a second image image the surface of the same object, wherein if at least one image segment appears in the second image, the surfaces output of the first image and the second image are the same.
7. Use of the method according to any one of the preceding claims 1 to 5, wherein the method is used to verify the authenticity of a product, wherein a first image and a second image image the surface of the same product, in particular the surface of an object, wherein if at least one image fragment appears in the second image, the surfaces output of the first image and the second image are the same.
8. An apparatus arranged to perform the method according to any one of the preceding claims 1 to 5.
9. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to claims 1 to 5.
10. A machine-readable storage medium having stored thereon the computer program according to claim 9.
Citation Information
Cited By
Method and device for ascertaining whether a specified image detail of a first image occurs in a second image
US20240395011A1
