Verification device and program

Abstract

To provide a technique capable of acquiring a collation area more reliably than simply acquiring a collation area by relative coordinates from a reference mark.SOLUTION: A registration image of a target object 10 is captured with a registration image photographing device 20 and registered in a server computer 50. A processor of a collation image photographing device 22 executes a program, thereby (a) acquiring a captured image including a collation area provided on the target object 10 and a reference position area serving as a reference for the position of the collation area, (b) detecting the reference position area from the captured image, (c) setting a surrounding image area within the captured image based on the reference position area and predetermined relative coordinates, and (d) detecting the collation area in the surrounding image area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a collation device and a program. [Background technology]

[0002] Patent Document 1 describes a system with few restrictions on image acquisition for determining the authenticity of a product using a captured image of a random pattern. Specifically, the system determines the authenticity of a product by comparing a random pattern in a specific area of ​​the product, and includes: a memory that stores feature amounts of the area of ​​the genuine product in a reference state of a mark at a specific position relative to the area; image acquisition means for photographing the target product including the mark and the area; image processing means that acquires conversion parameters that make the mark in the image the same as the mark in the reference state and corrects the area in the image using the parameters to an image that shows the area when the mark is in the reference state; extraction means that extracts feature amounts from the image of the area portion; comparison processing means that calculates similarity by comparing the feature amounts obtained by the extraction means with the feature amounts stored in memory; and determination means that determine whether the similarity exceeds a predetermined threshold.

[0003] Patent Document 2 describes a data acquisition device that includes an acquisition means for acquiring, as registration data, feature data representing features distributed in an area of ​​a predetermined size that is determined based on both the external shape of the object and a position defined by the position of printed information printed on the object, from an image including the object to be registered, and a storage means for storing the registration data acquired by the acquisition means as data for determining the identity of the object, wherein the acquisition means determines the position of the area of ​​the predetermined size based on both the external shape of the object and a position defined by the position of the printed information printed on the object, and acquires the feature data.In this way, if the position of the printed information on a certain object is shifted from a predetermined printing position, the data acquisition device determines the position of the area of ​​the predetermined size in a positional relationship that differs from the positional relationship between the position of the printed information on another object whose position of the printed information is not shifted from the predetermined printing position, and acquires the feature data.

[0004] Patent Document 3 describes an image processing method in which, in order to detect an image pattern to be searched for from photographed images of a product to be inspected that contain random noise, an image pattern identical to the search target is registered in advance as a template image, multiple images of the product to be inspected are obtained by photographing the product to be inspected multiple times at the same position, and a searched image is created by performing weighted averaging on the multiple photographed images, and pattern matching is performed on the searched image using the template image.In this image processing method, an object to serve as an inspection reference, which has the same size, shape, and pattern as the product to be inspected, is photographed in advance as an image to assist the weighted averaging, and an image to assist the weighted averaging, which has the same size and image pattern as the template image, is extracted and registered from the photographed image of this reference object, and the weighted averaging is performed by selecting from the multiple photographed images an image area that has the same coordinates as the photographed images and is the same size as the template image, shifting it by one pixel from the start point to the end point, and adding the weighted averaging auxiliary image to the selected image area, and pattern matching the weighted averaging processed image with the template image.

[0005] Patent Document 4 describes an individual identification device that includes a generation unit that generates a pattern on an object, an imaging unit that detects the generation of the pattern and images the generated pattern, and a housing that houses the generation unit and imaging unit and has an opening on its underside, the generation unit is configured to be able to move freely back and forth between a standby position and the generation position, and generates the pattern on the object through the opening when the housing moves from the standby position to the generation position while being placed on the object so that the underside of the housing is in contact with the object, the housing has a first detection means that detects that the generation unit is located at the generation position, the imaging unit detects the generation of the pattern based on the detection result of the first detection means and images the pattern on the object through the opening, the housing has a second detection means that detects that the generation unit is located at the standby position, and the imaging unit detects the generation of the pattern based on the detection result of the first detection means, and then takes an image when the generation unit moves from the generation position to the standby position based on the detection result of the second detection means. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-029678 [Patent Document 2] Patent No. 6751521 [Patent Document 3] Patent No. 4865204 [Patent Document 4] Patent No. 6813807 Summary of the Invention [Problem to be solved by the invention]

[0007] When a user takes a photo of the matching area using a mobile device and compares it with a registered image, it is necessary to reliably acquire the matching area, but if the user holds the mobile device while taking the photo, the position of the matching area may become unstable due to camera shake, etc. It is possible to prepare a reference mark separate from the matching area and specify the matching area using predetermined relative coordinates from the reference mark, but camera shake, etc. during shooting will change the relative positional relationship between the reference mark and the matching area, making it difficult to reliably acquire the matching area using this method alone.

[0008] An object of the present invention is to provide a technique that can acquire a matching area more reliably than when the matching area is simply acquired using relative coordinates from a reference mark. [Means for solving the problem]

[0009] A first aspect of the present invention provides a method for implementing a method of controlling a computer system, the method comprising: (a) acquiring a photographed image including a matching area provided on an object and a reference position area that serves as a reference for the position of the matching area; (b) detecting the reference position area from the captured image; (c) determining a position in the captured image based on the reference position area and predetermined relative coordinates; Virtual Set the surrounding image area, (d) detecting the matching region within the peripheral image region; It is a matching device.

[0010] A second aspect is the matching device according to the first aspect, further comprising a display unit, and the processor displays a guide indicating the surrounding image area on the display unit.

[0011] A third aspect is the verification device according to the second aspect, wherein the processor displays a guide indicating the verification area on the display unit.

[0012] In a fourth aspect, the processor further Image quality evaluation value and if the evaluation value is equal to or less than a threshold, repeats the processes (a) to (d) to acquire a match image for which the evaluation value exceeds the threshold.

[0013] A fifth aspect is a matching device according to any one of the first to fourth aspects, in which the processor, after setting the surrounding image area, determines the color or pattern of the background area of ​​the surrounding image area in the captured image, and if the color or pattern does not match a predetermined condition, repeats the processes (a) to (c).

[0014] A sixth aspect is the matching device according to any one of the first to fifth aspects, wherein the processor further acquires the relative coordinate data from an external server.

[0015] A seventh aspect is the verification device according to any one of the first to fifth aspects, wherein the processor further notifies a user when the reference position area is detected.

[0016] An eighth aspect is the matching device according to the fifth aspect, wherein the processor further notifies a user when the color or pattern matches a predetermined condition.

[0017] A ninth aspect is a matching device according to any of the first to eighth aspects, wherein the matching area has a specific shape such as an ellipse including a circle, or a polygon, and the processor detects the matching area by extracting the specific shape from within the surrounding image area.

[0018] A tenth aspect is the verification device according to any one of the first to eighth aspects, wherein the reference position area includes a QR code (registered trademark).

[0019] An eleventh aspect is a method for providing a computer processor with: (a) acquiring a photographed image including a matching area provided on an object and a reference position area that serves as a reference for the position of the matching area; (b) detecting the reference position area from the captured image; (c) determining a position in the captured image based on the reference position area and predetermined relative coordinates; Virtual Set the surrounding image area, (d) detecting the matching region within the peripheral image region; It is a program that executes this. [Effects of the Invention]

[0020] According to the first and eleventh aspects, the verification area can be acquired more reliably than when the verification area is acquired simply by using relative coordinates from the reference mark.

[0021] According to the second and third aspects, the user can further use the guide to easily photograph the verification area.

[0022] According to the fourth and fifth aspects, it is possible to take a comparison image with even better image quality.

[0023] According to the sixth aspect, it is further possible to acquire data of relative coordinates according to the object.

[0024] According to the seventh and eighth aspects, the user can easily visually check whether the photographing conditions are appropriate or not.

[0025] According to the ninth aspect, the matching region can be extracted more reliably.

[0026] According to the tenth aspect, the reference position area can be detected more reliably. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a system configuration diagram of an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of the collation image photographing device according to the embodiment. [Figure 3] 1A and 1B are plan views of an object including a hologram portion and a paper object according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating a peripheral area setting according to an embodiment. [Figure 5] 10 is a first explanatory diagram of a guide displayed on the output unit (display unit) of the collation image photographing device according to the embodiment. FIG. [Figure 6] 10 is a second explanatory diagram of a guide displayed on the output unit (display unit) of the collation image photographing device according to the embodiment. FIG. [Figure 7] 1 is a processing flowchart of an embodiment. [Figure 8] FIG. 10 is a diagram illustrating background color evaluation of a captured image according to an embodiment. [Figure 9] 10 is a detailed flowchart of a shape extraction process according to an embodiment. [Figure 10] 10A and 10B are explanatory diagrams showing the positional relationship between a peripheral image region, an ink portion, a registered image, and a match image according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The description will be made taking as an example an individual identification system that takes a surface image of an object and performs image matching between a registered image and a match image to uniquely identify the object.

[0029] An individual identification system is a technology that pre-registers a portion of an object's surface—specifically, an image of a size between 0.1 and several millimeters—as unique information about the object. The system then uniquely identifies the object being compared as the registered object, i.e., as genuine. This unique information can be, for example, a random pattern based on a fine pattern. A specific example of a random pattern based on a fine pattern is a matte finish. This matte finish is not limited to surface finishes such as frosted glass, but also includes matte finishes applied by processing metals or synthetic resins (e.g., plastics), as well as wrinkled patterns obtained by embossing, randomly woven fiber patterns, random fine dot patterns printed with ink containing glitter particles, and unevenness formed on sandblasted glass surfaces. Furthermore, it includes not only matte finishes formed unintentionally by chance, but also matte finishes intentionally formed for identification or matching purposes. In short, it is a random pattern that is difficult to control and form. Optically reading such random patterns and using them as information can be considered a type of "artifactometrics."

[0030] Here, we consider a case where a printing substrate with an uneven surface, such as a hologram or paper, is used as the printing substrate, and an ink portion with dispersed metal particles is printed on such an uneven printing substrate to create a random pattern.

[0031] 1 shows the system configuration of this embodiment. The matching system is made up of a registration image photographing device 20, a match image photographing device 22, and a server computer 50. The registration image photographing device 20 and the server computer 50, and the match image photographing device 22 and the server computer 50 are connected via a communication network.

[0032] A light source unit 21 such as an LED illuminates the object 10, and the light reflected from the object 10 is captured by a registration image capture device 20 to obtain a registration image. The registration image capture device 20 and the light source unit 21 may be configured as dedicated equipment for registration.

[0033] The irradiation angle φ of the light emitted from the light source unit 21 is set to a certain angle. The acquired registered image is transmitted to the server computer 50 and stored in the registered image DB 50b in the server computer 50.

[0034] On the other hand, a portable terminal such as a smartphone held by a user of the matching system serves as a matching image photographing device 22 to photograph the object 10. A light source unit 22a such as an LED mounted on the smartphone or the like illuminates the object 10, and a camera 22b mounted on the smartphone or the like photographs the light reflected from the object 10. The irradiation angle φ of the light irradiated from the light source unit 22a is set to be substantially the same as the angle φ under which the registered image was acquired. The reason for this is that, as described above, the random pattern of the ink portion changes depending on the direction of light irradiation, so the positional relationship between the light source unit 22a, camera 22b, and object 10 needs to be set to be substantially the same as the positional relationship when the registered image 16 was photographed.

[0035] The processor of the match image photographing device 22 performs a series of processes on the photographed image to extract the ink portion from the photographed image, and then cuts out a match image from within the ink portion as a match region, and transmits the match image to the server computer 50 via the communication network. The processing of the processor of the match image photographing device 22 will be described in more detail below.

[0036] The server computer 50 includes a matching unit 50a and a registered image DB 50b.

[0037] The registered image DB 50b is configured with a storage device such as a hard disk or SSD (Solid State Drive), and stores an identifier ID for uniquely identifying the object 10 and the registered image in association with each other.

[0038] The matching unit 50a is composed of a processor and stores the registered image received from the registered image photographing device 20 in the registered image DB 50b in association with the ID of the object 10. It also performs image matching between the match image received from the match image photographing device 22 and the registered image stored in the registered image DB 50b, and outputs the matching result to the match image photographing device 22. Specifically, the matching unit 50a reads the registered image from the registered image DB 50b, performs matching calculations with the match image, and calculates the similarity between the two images. The similarity calculation can use known algorithms such as feature matching using feature detection or template matching using image shading comparison. The calculated similarity is compared with a threshold value, and if it exceeds the threshold, it is determined that the two images match; if it does not exceed the threshold, it is determined that the two images do not match. The matching unit 50a transmits the matching result to the match image photographing device 22 via a communication network.

[0039] Image matching involves an error rate due to fluctuations in the input of the image sensor of the registration image capture device 20 or the match image capture device 22, quantization errors, and other factors. The error rate is made up of two components: the false rejection rate, which is the probability that a true value is judged as false, and the false acceptance rate, which is the probability that a false value is judged as true. There is a trade-off between the two, and as one decreases, the other increases. Therefore, the threshold is set to minimize the loss in the target to which the matching judgment is applied.

[0040] It is also possible to obtain a plurality of registered images by changing the direction of light irradiation and register them in the registered image DB 50b of the server computer 50, and perform image matching between these registered images and the match image.

[0041] 2 shows a block diagram of the main components of a match image photographing device 22, such as a smartphone. In addition to the light source unit 22a and camera unit 22b described above, the match image photographing device 22 includes a processor 22c, a ROM 22d, a RAM 22e, an input unit 22f, an output unit 22g, and a communication interface 22h.

[0042] The processor 22c reads out an application program stored in the ROM 22d, executes a series of processes using the RAM 22e as a working memory, extracts the ink portion from the image captured by the camera unit 22b, and cuts out a match image. The processor 22c transmits the cut-out match image to the server computer 50 via the communication I / F 22h. The processor 22c also receives a match result from the server computer 50 via the communication I / F 22h.

[0043] The input unit 22f is composed of a keyboard, touch switches, etc., and is operated by the user to start an application program.

[0044] The output unit 22g is configured with a liquid crystal display, an organic EL display, or the like, and functions as a display unit, displaying a preview image when photographing the object 10. The output unit 22g may also display a guide when photographing the object 10 in response to a control command from the processor 22c. The guide will be described in more detail below. Furthermore, the output unit 22g displays the matching result received from the server computer 50 in response to a control command from the processor 22c. The matching result is either "match" or "mismatch," but other messages related to the matching may also be displayed.

[0045] FIG. 3 shows a plan view of the object 10 in this embodiment.

[0046] 3(a) shows a case where a hologram label sticker is used as the target object 10. The hologram label sticker includes hologram portions 12a and 12b, a QR code 13, and an ink portion .

[0047] The hologram portion 12a is formed in approximately the left half of the label sticker and forms a hologram pattern.

[0048] The hologram section 12b is formed on roughly the right half of the label sticker, has a matte finish, and emits rainbow colors that change depending on the elevation angle. Here, the "elevation angle" refers to the angle between the LED light source 22a, the target object 10, and the camera section 22b in Figure 1. It is desirable that this elevation angle coincides with the angle φ in Figure 1.

[0049] The QR code 13 is formed on the matte-finished hologram 12b. Various information about the label sticker is printed as a QR code on the QR code 13. In this embodiment, the QR code 13 is printed at a fixed position relative to a printing alignment mark (not shown). The ink portion 14 is printed at a fixed position relative to the printing alignment mark in a separate process. Therefore, the QR code 13 and the ink portion 14 are printed with a generally fixed relative positional relationship, and the QR code 13 can also function as an alignment mark and a reference position area for extracting the ink portion 14.

[0050] The ink portions 14 are gravure printed on the matte-finished hologram 12b with gravure ink containing silver particles, and are polygonal in shape. In the figure, the ink portions 14 are printed in squares at regular intervals below the QR code 13. These ink portions 14 are random pattern areas, and are matching areas that should be photographed and extracted by the matching image photographing device 22. The shape of the ink portions 14 may be an ellipse (including a circle) other than a polygon.

[0051] 3(b) shows a case where a paper label sticker is used as the other object 10. The paper label sticker includes a paper portion 11, a QR code 13, and an ink portion 14.

[0052] The QR code 13 is formed on the paper portion 11. Various information about the label sticker is printed as a QR code on the QR code 13. The QR code 13 is printed with its relative position relative to the ink portion 14 fixed in advance. Therefore, the QR code 13 can also function as an alignment mark for extracting the ink portion 14.

[0053] The ink portion 14 is toner-printed on the paper portion 11 with toner ink containing silver particles, and has a polygonal shape. In the figure, the ink portion 14 is printed in the form of a square at regular intervals below the QR code 13. This ink portion 14 is a random pattern area, and is a matching area to be photographed and extracted by the matching image photographing device 22.

[0054] 4 shows a schematic diagram of the positional relationship between the QR code 13 and the ink portion 14. If a mobile terminal such as a smartphone is used as the match image photographing device 22, the user must manually photograph the image, which makes the position of the ink portion 14 in the photographed image unstable. Since the match image to be sent to the server computer 50 must be an image of an accurate position, the QR code 13 formed near the ink portion 14 in a predetermined relative positional relationship is used as a positioning mark.

[0055] 4(a) schematically shows the process of identifying the position of the ink portion 14 using the relative positional relationship between the QR code 13 and the ink portion 14. To extract the ink portion 14, the QR code 13 is first detected. Once the QR code 13 is detected, the position of the ink portion 14 is identified using the known relative positional relationship between the QR code 13 and the ink portion 14 (in the figure, this relative positional relationship is shown as a vector). Then, the square ink portion 14 is extracted from the captured image at the identified position.

[0056] However, variations in the printing position may cause variations in the relative positional relationship between the QR code 13 and the ink portion 14. Furthermore, the relative positional relationship between the QR code 13 and the ink portion 14 may fluctuate due to hand shake during shooting, etc. Therefore, it is difficult to reliably and stably acquire the ink portion 14 as the verification area using only the QR code 13 as an alignment mark and information on the known relative positional relationship.

[0057] Therefore, as shown in Figure 4(b), when the QR code 13 is detected, a peripheral image region 15 is set that is larger than the known size of the ink portion 14 and that encompasses the ink portion 14 based on a second relative positional relationship that has been set in advance. The shape of the peripheral image region 15 is arbitrary, but it may be, for example, a square that is similar to the shape of the ink portion 14, as shown by the dashed dotted line in Figure 4(b). The peripheral image region 15 is set to a shape and size that can reliably encompass the ink portion 14 even if there is variation in the relative positional relationship between the QR code 13 and the ink portion 14. As an example, the peripheral image region 15 may be set to a shape similar to the shape of the ink portion 14 and 50% larger than the ink portion 14, but is not limited to this.

[0058] Once the surrounding image area 15 is set based on the positional relationship with the QR code 13 as the alignment mark, the processor 22c of the collation image photographing device 22 extracts the ink portion 14 of a known shape (square in this case) within this surrounding image area 15.

[0059] The peripheral image area 15 is set as an internal process by the processor 22c of the match image photographing device 22, but the processor 22c may display the set peripheral image area 15 on the output unit 22g so that the user can see it. The peripheral image area 15 displayed on the output unit 22g functions as a guide for the user to the ink portion 14.

[0060] Fig. 5 shows an example of a guide displayed on the output unit 22g of the match image photographing device 22. Fig. 5(a) shows an example of a guide frame 15g indicating the surrounding image area 15 set by the processor 22c. The surrounding image area 15 is set as an area of ​​a predetermined shape and size that is in a predetermined second relative positional relationship with the QR code 13, using the previously detected QR code 13 as an alignment mark. The guide frame 15g is displayed superimposed on the preview display of the output unit 22g in accordance with the surrounding image area 15 set in this manner.

[0061] That is, a preview of the captured image of the object 10 is displayed on the output unit 22g, and a guide frame 15g is displayed based on the QR code 13 detected in this preview display. Using this guide frame 15g as a guide, the user can adjust the distance between the collation image photographing device 22 and the object 10 as appropriate, and adjust so that the ink portion 14 is positioned within the guide frame 15g.

[0062] Note that it is desirable that the distance between the match image photographing device 22 and the object 10, more specifically the distance (photographing distance) between the camera unit 22b of the match image photographing device 22 and the object 10, be close to the distance (photographing distance) when the registered image was acquired. Therefore, the processor 22c may measure the photographing distance between the camera unit 22b and the object 10, and display the guide frame 15g on the output unit 22g when the photographing distance approaches the desired photographing distance. This allows the user to hold the match image photographing device 22, such as a smartphone, and gradually approach the object 10 with the camera unit 22b facing the object 10, until the guide frame 15g is previewed and the position of the match image photographing device 22 is adjusted so that the ink portion 14 is positioned within the guide frame 15g. Because the size of the ink portion 14 is known and the size of the peripheral image area 15 is set according to the size of the ink portion 14, the size of the guide frame 15g that is previewed when the photographing distance approaches the desired photographing distance is also calculated by the processor 22c.

[0063] 5(b) shows another example of a guide displayed on the output unit 22g of the match image photographing device 22. In addition to the guide frame 15g indicating the surrounding image area 15, a guide frame 13g indicating the QR code 13 is displayed on the output unit 22g, and a guide frame 14g indicating the ink portion 14 is also displayed on the output unit 22g. The guide frame 13g is displayed along the outline of the QR code 13 when the QR code 13 is detected. Furthermore, the guide frame 14g is displayed after the guide frame 15g is displayed or together with the display of the guide frame 15g, as a candidate area where the ink portion 14 exists within the guide frame 15g. The guide frames 13g, 14g, and 15g each guide a different target, and therefore may be displayed in different colors or patterns so as to be distinguishable from one another.

[0064] As described above, by setting the surrounding image area 15 based on the QR code 13 and superimposing the guide frame 15g indicating this surrounding image area 15 on the preview display of the captured image, the user can align the matching image camera 22 while visually checking the guide frame 15g.

[0065] On the other hand, with regard to the elevation angle φ (see FIG. 1) of the collation image photographing device 22 relative to the object 10, when the object 10 is as shown in FIG. 3(a), it is possible to utilize the color change of the matte-finished hologram portion 12b on which the ink portion 14 is formed. That is, since the rainbow color of the matte-finished hologram 12b changes depending on the elevation angle, the elevation angle φ can be adjusted by adjusting the color of the hologram portion 12b around the ink portion 14, or more specifically, the color of the peripheral image area 15 excluding the ink portion 14, so that it becomes the desired color (or pattern).

[0066] Specifically, the processor 22c detects a color pattern in the peripheral image region 15 in the captured image, determines whether or not it is a desired color pattern according to the elevation angle φ, and displays the determination result as a guide on the output unit 22g.

[0067] FIG. 6 shows another example of a guide displayed on the output unit 22g of the collation image capture device 22. FIG. 6(a) shows an example of a guide when the coloring pattern of the hologram unit 12b in the peripheral image region 15 is a desired coloring pattern (desired rainbow colors). In addition to the guide frames 13g, 14g, and 15g, a mark 24 of a specific color (e.g., green) indicating a desired elevation angle is displayed near the guide frame 15g indicating the peripheral image region 15. By visually checking the guide frames 13g, 14g, and 15g and the mark 24, the user can easily confirm that the capture position of the collation image capture device 22 is the desired position. Note that in FIG. 6(a), a guide indicating a desired specular reflection position of the light source unit 22a may be displayed in addition to the mark 24. The user can more reliably adjust the elevation angle φ by adjusting the specular reflection position of the light source unit 22a in the captured image so that it approximately matches this mark. This is because the elevation angle φ is determined by the positional relationship between the light source unit 22a, the target object 10, and the camera unit 22b, as shown in FIG.

[0068] FIG. 6(b) is an example of a guide when the coloring pattern of hologram portion 12b in peripheral image region 15 is not the desired coloring pattern (desired rainbow color). Guide frames 13g and 15g are displayed, but mark 24 is not. Instead of mark 24, message 25 is displayed indicating that elevation angle φ is inappropriate or indicating the direction to adjust elevation angle φ to the desired angle. In FIG. 6(b), the rainbow color in peripheral image region 15 corresponds to a coloring pattern that appears when the head side of matching image photographing device 22 (in the case of a smartphone, the side where light source unit 22a and camera unit 22b are provided) is lowered, and message 25 reading "Head UP" is displayed to urge the user to raise the head side.

[0069] In this way, by displaying the mark 24 or message 25 on the output section 22g, the user is prompted to pay attention to the color and pattern around the ink section 14, making it easier to match it with the sample shown in the manual, etc., thereby shortening the time required to capture a good image of the ink section 14.

[0070] In FIG. 6(b), the guide frame 14g of the ink portion 14 is displayed in gray, but this is in consideration of the fact that the color around the ink portion 14 is inappropriate. The processor 22c displays all guides when shooting begins, and changes the color of the guide frame 14g and changes whether or not the mark 24 or message 25 is displayed during shooting. Furthermore, in consideration of ease of positioning by the user, the processor 22c may gradually change the display so that it displays the guide frame 13g and the guide frame 15g after detecting the QR code 13 and setting the peripheral image area 15, and then displays the guide frame 14g of the ink portion 14 when it is determined that the color in the peripheral image area 15 is the desired color.

[0071] Next, the processing of the processor 22c in this embodiment will be described in detail using the case where the object 10 is as shown in FIG. 3(a) as an example.

[0072] 7 is a flowchart showing the processing of the processor 22c, which is realized by reading and executing a program stored in the ROM 22d or the like.

[0073] First, the processor 22c acquires a photographed image captured by the camera unit 22b (S11).

[0074] Next, an alignment mark is detected from the acquired captured image (S12). The alignment mark is, for example, a QR code 13, and the QR code 13 is identified from the captured image by detecting the cutout symbol, timing pattern, alignment pattern, and format information contained in the QR code 13.

[0075] When the alignment mark and the QR code serving as the reference position area are detected (YES in S12), the peripheral image area 15 is set using second relative positional relationship data predetermined and stored in memory, with the position of the QR code 13 as the reference. Alternatively, the processor 22c may access the server computer 50 and acquire the second relative positional relationship data (relative coordinate data) from the server computer 50. At this time, the ID of the object 10 obtained by reading the QR code 13 is transmitted to the server computer 50, whereby the relative positional relationship data corresponding to the object 10 can be acquired from the server computer 50. Then, a guide frame 15g indicating the peripheral image area 15 is displayed on the output unit 22g, and a peripheral image corresponding to this peripheral image area 15 is acquired from the captured image (S13). The acquired peripheral image includes an image of the ink portion 14 as a random pattern, and is therefore referred to as a random pattern peripheral image.

[0076] Once the random pattern peripheral image is acquired, it is determined whether the background color of the ink portion 14 included in this random pattern peripheral image is appropriate (S14). That is, the coloring pattern of the background color is compared with a predetermined desired coloring pattern stored in memory, and if the two match, it is determined to be appropriate. If the two do not match, it is determined to be inappropriate. Specifically, the center of gravity of a specific color included in the background color is calculated and compared with the center of gravity corresponding to the desired coloring pattern. If it is determined that the background color is inappropriate (NO in S14), a message 25 shown in FIG. 6(b) is displayed on the output unit 22g, and the process from S11 onward is repeated.

[0077] Figure 8 shows a schematic diagram of the process of S14. Figure 8(a) shows an example of a case where the background color of the random pattern peripheral image 28 around the ink portion 14 is appropriate. Figures 8(b) and 8(c) show examples of a case where the background color of the random pattern peripheral image 28 around the ink portion 14 is inappropriate.

[0078] The background color of the random pattern peripheral image 28 is a unique change in the matte-finished hologram unit 12, i.e., the rainbow coloring changes depending on the elevation angle; in other words, the characteristics of the reflected light change depending on the direction of light irradiation. While it is possible to estimate the elevation angle using the shape of the QR code 13 used as an alignment mark, the relatively large size required places restrictions on the target object 10 and its design. Furthermore, under conditions where the target object is within the field of view of the camera unit 22b and is positioned appropriately relative to the light source unit 22a, the area available for estimating the elevation angle is small, resulting in reduced accuracy.

[0079] In contrast, in matte hologram section 12b, the rainbow color generation varies depending on the elevation angle, so the elevation angle can be reliably estimated by calculating the center of gravity of a specific color (e.g., red, yellow, light blue) in random pattern peripheral image 28 by binarizing the RGB-decomposed image and determining whether it falls within a preset threshold. In the case of a smartphone or the like in which light source section 22a and camera section 22b are integrally mounted, determining whether the elevation angle is appropriate can be said to be synonymous with determining whether the smartphone is in the appropriate orientation.

[0080] Returning to FIG. 7 again, if the background color is determined to be appropriate (YES in S14), the mark 24 shown in FIG. 6(a) is displayed on the output unit 22g, and the shape of the ink portion 14 (a square in this embodiment) is extracted from the acquired random pattern peripheral image 28 (S15).

[0081] FIG. 9 shows a detailed flowchart of shape extraction.

[0082] This processing flowchart aims to obtain the coordinates of the four vertices of the square ink portion 14 from the random pattern peripheral image 28, and is broadly divided into three processes: a binary image generation process (S1), a rectangular edge extraction process (S2), and a vertex coordinate estimation process (S3).

[0083] <Binarized image generation processing> First, the binary image generation process (S1) will be described.

[0084] In this process, first, a smoothing process and a density difference enhancement process are simultaneously performed on the original image of the random pattern peripheral image 28 (S101). Simply performing a smoothing process on the original image will blur the shape of the ink portion 14. Furthermore, simply performing a density difference enhancement process on the original image will emphasize the unevenness of the ink portion 14, but will also emphasize the unevenness of the hologram portion 12b, making it impossible to extract the ink portion 14.

[0085] Therefore, smoothing and contrast enhancement are simultaneously performed on the original image to remove the irregularities in the hologram portion 12b and the ink portion 14, thereby distinguishing the ink portion 14 from the hologram portion 12b. Specifically, a mean-shift filter can be used to simultaneously perform the smoothing and contrast enhancement processes. The mean-shift filter is implemented by processor 22c. The mean-shift filter is a filter that fills similar colors within a specified pixel space with the same color. This causes the silver ink portion 14 to approach the same color, and because the boundary between the rainbow background of hologram 12b and the silver ink portion 14 is a different color region, the contrast at the boundary between hologram portion 12b and the ink portion 14 is emphasized while maintaining the shape of the ink portion 14.

[0086] There are filtering methods that preserve edges and perform smoothing processing, such as bilateral filters, but the inventors have confirmed that these cannot remove noise in the hologram portion 12b and ink portion 14. By using a mean-shift filter, edges are preserved and smoothing is performed for each color using the color difference between the hologram portion 12b and ink portion 14, making it possible to remove noise without losing edges.

[0087] The mean-shift filter first searches for the center of gravity of the color distribution of the original image. That is, it calculates the center of gravity coordinates (xc,yc) and color (rc,gc,bc) of the color space area of ​​radius sr centered on the color (r,g,b) of a pixel (x,y), and then searches for the center of gravity under the following conditions: where sp is the radius of the search area. Conditions: |x-xc|≦sp, |y-yc|≦sp, ||(r,g,b)-(rc,gc,bc)||≦sr

[0088] And when the above conditions are met, (x,y,r,g,b)=(xg,yg,rc,gc,bc) Then, the centroid search is performed again. The above centroid search process is repeated.

[0089] Then, the color space distance ε and the number of repetitions n are set in advance, and it is determined whether or not the following condition is met, and if so, the process ends. Condition: Meet n iterations or |x-xc|+|y-yc|+(r-rc) 2 +(g-gc) 2 +(b-bc) 2 <ε

[0090] After the centroid search process is completed, the image is smoothed using the centroid value of the color space. That is, after the polar search is completed, each pixel in the space is set to the centroid value of that color space. Then, the edges are clarified using the Gaussian pyramid and the threshold value sr.

[0091] The mean-shift filter performs smoothing processing using distance differences in color space, and is therefore an effective smoothing process when there is a difference in color space distance between the foreground and background. For this reason, it is an effective process for an original image in which achromatic ink portions 14 exist in the foreground and chromatic hologram portions 12b exist in the background.

[0092] In the mean-shift filter, the performance of the smoothing process and the shading enhancement process can be controlled using the color space radius sr and the pixel space radius sp as the main parameters. Therefore, by adjusting these parameters, the ratio of the smoothing process to the shading enhancement process can be adjusted. Specifically, (1) The pixel space radius sp specifies the search range for pixels to be smoothed (filled in), so Large SP → Wide search range SP is small → Search range is narrow Note that if sp is set too large, it will take a long time to process, so this should be taken into consideration. (2) The range of similar colors to be filled with the same color is determined by the color space radius sr. High SR → Slightly different colors are recognized as the same color Small sr → Similar colors are recognized as the same color In this embodiment, the parameters sp and sr of the mean-shift filter can be adjusted as follows: (sp,sr)=(10,30) It is set as follows:

[0093] After smoothing and contrast enhancement are simultaneously performed on the original image (S101), an additional contrast enhancement process is performed on the areas where contrast cannot be obtained by the process of S101 (S102).

[0094] If there is not a sufficient difference in color space distance between the background color and the ink portion 14, the foreground and background will blend together, and there may be areas where the contrast enhancement is insufficient with only the processing of S101. Therefore, by further executing the contrast enhancement processing, the shape of the ink portion 14 can be extracted more stably.

[0095] Specifically, the image processed in S101 is decomposed into RGB, and contrast enhancement processing is performed in each RGB color space. This means flattening the brightness histogram within the image. Then, a Sobel filter is applied to each of the vertical and horizontal RGB color images to extract the edge gradient. Note that the gradient value calculated by the Sobel filter is not 8 bits (256 levels), so it can be normalized to 8 bits. The normalization method is to take the absolute value of the gradient image and replace all pixel values ​​above 255 with 255. This makes it possible to obtain the edge gradient without relying on external noise.

[0096] After performing additional shading enhancement processing (S102), noise reduction processing is performed using the HSV color space (S103). The HSV color space is a color space consisting of three components: hue, saturation (saturation / chroma), and brightness (value / brightness).

[0097] When the rough shape of the ink portion 14 is extracted in S102, noise occurs at the boundary between the white and light blue of the hologram portion 12. In particular, because the gradient between the white and light blue of the R space image is large, edge-like noise occurs when a Sobel filter is applied. Therefore, this noise is removed using the HSV color space. Specifically, (1) HSV decomposition of the processed image 13 in S101 (2) Binarize the S image (3) Apply vertical and horizontal Sobel filters to the binary image. (4) OR composite the vertical and horizontal Sobel image and the black and white inverted binary image of the H image.

[0098] After the process of S103 is performed, a binarized image is created (S104). That is, a total of six vertical and horizontal gradient images of the R, G, and B images are binarized. The binarization threshold may be set differently for R, G, and B. Then, a total of six binarized images of the vertical and horizontal components and RGB color components are OR-combined.

[0099] <Rectangle edge extraction processing> Next, the rectangular edge extraction process will be described.

[0100] After creating a binarized image in S104, the edges of the polygons that make up the square ink portion 14 are obtained from this binarized image (S105). Specifically, this is edge extraction processing using a probabilistic Hough transform. Note that the probabilistic Hough transform is an optimized version of the Hough transform, and instead of using all pixels, calculations are performed by randomly selecting enough points from the image to detect lines. A (non-probabilistic) Hough transform can also be used in S104. However, this has the drawbacks of being difficult to tune the parameters and being too sensitive to rectangular edges in the binarized image.

[0101] After obtaining the polygon edges (S105), a process for removing unnecessary edges is performed (S106). That is, lines that are not rectangular edges (edges) are removed from the lines extracted by the probabilistic Hough transform. Specifically, methods are used such as removing lines with a slope of a certain value or more, lines that are long relative to the size of the matching area 14, and lines where the angle at which vertical and horizontal lines intersect is within a certain angle, or removing lines that are tangent to the image frame. Alternatively, unnecessary edges may be removed by using a color space to extract edges with a color that resembles a rectangle.

[0102] <Vertex coordinate estimation process> After the rectangular edge extraction process (S2) is completed, the vertex coordinate estimation process (S3) of the square ink portion 14 is executed.

[0103] In this process, the centroid coordinates of the intersections of the edges are calculated from the image obtained by removing the unnecessary edges in S106 (S107). That is, instead of the intersections of each edge, the centroid coordinates of a group of intersections within a certain neighborhood are calculated. The intersections of vertical and horizontal lines are calculated for the processed image 23 by solving one-dimensional simultaneous equations. However, because the edge width of the binary image after OR compositing is 2 to 3 pixels, multiple lines are extracted for the same edge using the probabilistic Hough transform. As a result, multiple intersections exist near a certain coordinate. Since these intersections are likely to represent the same vertex, the centroid coordinates of the group of intersections are obtained and redefined as the vertices of the shape of the ink portion 14.

[0104] In the process of calculating the barycentric coordinates of a group of intersections, first, multiple intersections within a certain neighborhood are combined into one by dilation. The dilation process is a process in which, if there are white pixels in the surrounding pixels of a certain pixel, these pixels are converted to white pixels, thereby successively expanding the white pixels. Next, labeling is performed on each dilated point group. Then, the barycentric coordinates of each labeled point group are calculated. After calculating the barycentric coordinates in this way, the calculated barycentric coordinates are set as vertex candidates.

[0105] Since there are four vertices in the square ink portion 14, four vertex candidates can be set. When setting the vertex candidates, known shape characteristics of the ink portion 14, i.e., the lengths of the sides and diagonals, can be used as conditions. If there are multiple pairs of vertices that satisfy the conditions, the most plausible pair of vertices is selected. For example, in the square ink portion 14, the condition that the lengths of the four sides are equal can be used, and the pair of vertices that has the smallest variance in the side lengths can be set as the pair of vertices.

[0106] Then, it is determined whether all the vertices of the ink portion 14 have been acquired (S108). In the case of a square ink portion 14, it is determined that all the vertices have been acquired when four vertices have been acquired. If all the vertices cannot be acquired, this means that not all the sides of the ink portion 14 have been extracted, and so next, a process of complementing the missing sides is performed (S109).

[0107] In the edge interpolation process, it is determined whether the three sides that make up a square ink portion 14 have been extracted. Normally, when the ink portion 14 is printed on a hologram portion 12b as a printing substrate, extraction of that side may fail if the red background color of the hologram portion 12 overlaps with the red foreground color. In other words, this occurs when the difference in color space distance between the background and foreground is small. Therefore, it is first determined whether the three sides have been extracted. The selection of the three sides can be estimated from the known shape characteristics of the ink portion 14, i.e., the edge lengths and positions.

[0108] When three sides have been extracted, the length x of the side without an opposite side is calculated from the center of gravity coordinates that have already been calculated. Then, a new parallel side is drawn at a distance of the side length x. Specifically, let a, b, c, and d be the four sides that make up a square, with a and c being opposite sides and b and d being opposite sides. When only the three sides a, b, and c are extracted, a side parallel to b is drawn at a distance x away from b, called d. It is possible to estimate two parallel sides, one on each side of b, that are distance x away from b, but since one of them does not exist in the image, side d can be drawn uniquely. This completes the missing side.

[0109] After the missing edges are complemented, the coordinates of the centers of gravity of the intersections are calculated again to obtain the coordinates of all vertices.

[0110] After the missing edges are complemented, the threshold may be lowered on the binarized image obtained in S104, and the probabilistic Hough transform may be performed again to re-obtain the edges, and the edges thus obtained may be integrated with the complemented edges, and the vertex coordinate estimation process (S3) may be performed again.

[0111] 7, once the shape is extracted by obtaining the coordinates of the four vertices of the ink portion 14 as described above (YES in S15), the processor 22c obtains a random pattern image by cutting out the collation image based on the coordinates of these four vertices (S16).Then, the processor 22c evaluates the image quality of the obtained random pattern image, i.e., determines whether the image quality of the random pattern image is good (S17).

[0112] Specifically, whether the image quality is good or not can be determined by evaluating the following index values ​​and determining whether these index values ​​exceed a threshold value. (1) Are the position, size, and angle of the square appropriate? (2) Degree of blur (standard deviation of Laplacian filter value) (3) Degree of blur (maximum and minimum difference in standard deviation of Sobel filter values ​​in four directions) (4) Brightness (average brightness) (5) Randomness (a quarter-sized portion of the center of the image is cut out, and the correlation value between each coordinate of the image and an image of the same size as the starting point is calculated. The correlation value is calculated by subtracting the average value from the maximum value of the group of correlation values ​​and dividing the result by the standard deviation.) (6) Degree of light source deviation (aspect ratio of image brightness gradient = gradient when the average brightness values ​​of the same row are linearly approximated in the column direction / gradient when the average brightness values ​​of the same column are linearly approximated in the row direction)

[0113] It is possible to determine whether the image quality is good or not by using any one of these index values ​​or any combination of multiple index values, for example, by using (1) and (6), or by using (1), (5), and (6).

[0114] The processes of S11 to S17 are automatically repeated until a predetermined upper limit number N of images are acquired, or until a timeout occurs, or until the user interrupts the photographing operation (S18). Then, a matching process is performed using the upper limit N of random pattern images acquired, or multiple random pattern images acquired before the timeout occurs or photographing is interrupted (S19).

[0115] In the matching process, a matching request is sent to the server computer 50, attaching the acquired maximum N or multiple random pattern images. The matching unit 50a of the server computer 50 matches the received random pattern images with the registered images and returns the matching results to the matching image photographing device 22. The processor 22c receives the matching results from the server computer 50 and displays them on the output unit 22g.

[0116] As described above, in this embodiment, the peripheral image area 15 is set based on the QR code 13 as an alignment mark, and the ink portion 14 is extracted from within this peripheral image area. Therefore, even if the relative positional relationship between the QR code 13 and the ink portion 14 varies, the ink portion 14 can be extracted stably.

[0117] 10 shows an ink portion 14, a surrounding image region 15 set for the ink portion 14, and a registration and matching region 16 cut out from the ink portion 14 in this embodiment. Peripheral image area 15 → Extraction of shape of ink portion 14 → Cutting out match image 16 The area is gradually identified in this order.

[0118] Furthermore, in this embodiment, a determination is made as to whether the background color (or pattern) of the surrounding image area 15 is appropriate, and the shape of the ink portion 14 is extracted based on the result, but the color (or pattern) determination process is capable of high-speed processing although it has relatively lower accuracy compared to the shape extraction process, so it is possible to first determine whether the image is being taken in an appropriate posture, and if it is not appropriate, to take the image again, thereby increasing the frequency of image capture, and increasing the frequency of image capture can increase the probability of obtaining an image of the ink portion 14 with appropriate image quality within a certain period of time. In this sense, the surrounding image area 15 has the function of specifying the position of the ink portion 14, as well as the function of defining the determination area when determining whether the background color (or pattern) around the ink portion 14 is appropriate.

[0119] The above description has been given taking the example of the object 10 shown in Fig. 3(a), but if the printing substrate of the object 10 is paper 11 as shown in Fig. 3(b), instead of the process of S14 in Fig. 7, the QR code 13 can be used as a calibration patch, the brightness can be corrected using black or the like in the QR code 13, and then it can be determined whether the vertical brightness gradient in the QR code 13 is within an appropriate range. The vertical brightness gradient can be defined as the gradient when the average brightness values ​​in the same column are linearly approximated in the row direction.

[0120] Furthermore, in this embodiment, the hologram portions 12a and 12b are exemplified, but a polarizing sheet may be used instead.

[0121] In addition, in this embodiment, multiple random pattern images with good image quality are acquired and sent to the server computer 50 for comparison, but it is also possible to select one random pattern image with the best image quality and send it to the server computer 50 for comparison.

[0122] Furthermore, in this embodiment, the QR code 13 is used as the alignment mark and reference position area, but instead, the edge of the ink portion 14 itself may be used as the alignment mark and reference position area, and the surrounding image area may be set from there based on a third relative positional relationship. In this case, the surrounding image area is set within the ink portion 14, rather than being set so as to include the ink portion 14 as shown in Figure 4(b). This is because the registered image registered in the registered image DB 50b is not the entire ink portion 14, but only a partial area of ​​the ink portion 14.

[0123] Furthermore, the processor 22c in this embodiment refers to a processor in a broad sense, and includes a general-purpose processor (e.g., a CPU (Central Processing Unit) and a dedicated processor (e.g., a GPU (Graphics Processing Unit)), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a programmable logic device, and the like). Furthermore, the operations of the processor in this embodiment may not only be performed by a single processor, but may also be performed by multiple processors located in physically separate locations working together. Furthermore, the order of the operations of the processor is not limited to the order described in this embodiment, and may be changed as appropriate. [Explanation of symbols]

[0124] 10 Object, 12a, 12b Hologram section, 13 QR code, 14 Ink section, 15 Peripheral image area, 20 Registration image photographing device, 22 Matching image photographing device, 22a Light source section, 22b Camera section, 50 Server computer.

Claims

1. The processor executes a program to (a) acquiring a photographed image including a matching area provided on an object and a reference position area that serves as a reference for the position of the matching area; (b) detecting the reference position area from the captured image; (c) setting a virtual peripheral image area within the captured image based on the reference position area and predetermined relative coordinates; (d) detecting the matching region within the surrounding image region; Collation device.

2. Further comprising a display unit, The processor: a guide indicating the peripheral image area is displayed on the display unit; The verification device according to claim 1 .

3. The processor: a guide indicating the collation area is displayed on the display unit; The verification device according to claim 2 .

4. The processor further comprises: calculating an evaluation value of the image quality of the detected matching area image; When the evaluation value is equal to or less than the threshold, the processes (a) to (d) are repeated to obtain a match image whose evaluation value exceeds the threshold. The collation device according to any one of claims 1 to 3.

5. The processor: After the peripheral image area is set, the color or pattern of the background area of ​​the peripheral image area in the photographed image is determined, and if the color or pattern does not match a predetermined condition, the processes (a) to (c) are repeated. The collation device according to any one of claims 1 to 4.

6. The processor further acquires the relative coordinate data from an external server. The collation device according to any one of claims 1 to 5.

7. The processor further notifies a user when the reference position area is detected. The collation device according to any one of claims 1 to 5.

8. The processor further notifies a user when the color or pattern matches a predetermined condition. The verification device according to claim 5 .

9. the matching area has a specific shape such as an ellipse including a circle, or a polygon; The processor: detecting the matching area by extracting the specific shape from within the peripheral image area; The collation device according to any one of claims 1 to 8.

10. The reference position area includes a QR code. The collation device according to any one of claims 1 to 8.

11. The computer processor (a) acquiring a photographed image including a matching area provided on an object and a reference position area that serves as a reference for the position of the matching area; (b) detecting the reference position area from the captured image; (c) setting a virtual peripheral image area within the captured image based on the reference position area and predetermined relative coordinates; (d) detecting the matching region within the surrounding image region; A program that makes it happen.

Images