Image processing method, non-transitory recording medium, and image processing system

Abstract

The present disclosure provides an image processing method, a non-transitory recording medium, and an image processing system. The present disclosure aims to speed up processing. In the k+1th image search, from the object image of the k+1th compression rate smaller than the kth compression rate, the k+1th image search is prepared, and the k+1th candidate image is explored from the k+1th image search. The candidate image is added to the new candidate of the extracted image. The image processing method includes search processing, acquisition processing, and sub-exploration processing. The search processing performs multiple image searches. The acquisition processing acquires correlation information indicating the correlation between the Nth candidate image and the sub-candidate image. The sub-exploration processing obtains the sub-candidate image based on the Nth candidate image obtained by the Nth image search and the correlation information obtained by the acquisition processing, and adds the sub-candidate image to the prepared candidate for the Mth image search.

Description

Technical Field

[0001] This disclosure generally relates to image processing methods, non-transitory recording media, and image processing systems, and more specifically, to image processing methods, non-transitory recording media, and image processing systems for exploring features of an image containing a model image from an image of an object. Background Technology

[0002] The image inspection apparatus described in Reference 1 (JP2012-178106A) uses a pyramid algorithm for image matching to detect which parts of the inspection image (model image) are similar to the object image. In the pyramid algorithm, image matching is performed between the compressed object image and the compressed inspection image to determine the parts with high image similarity (hereinafter referred to as "candidate regions"). Then, for the image with a lower compression ratio, image matching is performed targeting the areas surrounding the candidate regions. Because the pyramid algorithm does not require image matching of the entire low-compression image, the processing is accelerated.

[0003] In the image inspection apparatus described in Document 1, it is desirable to further increase the processing speed.

[0004] The purpose of this disclosure is to provide an image processing method, a non-transitory recording medium, and an image processing system that enable high-speed processing. Summary of the Invention

[0005] The image processing method disclosed herein involves multiple image searches to explore images containing features of a model image from object images, i.e., extracting images. K is set as a natural number variable. N and M are set as natural number constants, where M is greater than N. In the first image search, a first candidate image is explored from the object images with a first compression ratio, and the first candidate image is added to the candidates for the extracted images. The first candidate image contains features of the model image with the first compression ratio. In the (k+1)th image search, preliminary candidates for the (k+1)th image search are selected from the object images with a compression ratio smaller than the kth compression ratio. The preliminary candidates for the (k+1)th image search include the candidates for the extracted images obtained through the kth image search. Further, in the (k+1)th image search, a (k+1)th candidate image is explored from the preliminary candidates used in the (k+1)th image search, and the (k+1)th candidate image is added to the new candidates for the extracted image. The (k+1)th candidate image contains features of the model image with the (k+1)th compression ratio. The image processing method includes a search process, an acquisition process, and a sub-exploration process. The search process performs multiple image searches. The acquisition process acquires relevant information representing the correlation between the Nth candidate image and the sub-candidate image. The sub-exploration process acquires the sub-candidate image based on the Nth candidate image obtained through the Nth image search and the relevant information acquired through the acquisition process, and adds the sub-candidate image to the preliminary candidates used in the Mth image search.

[0006] The non-transitory recording medium involved in one aspect of this disclosure is a non-transitory recording medium that records a program for causing one or more processors to execute the image processing method.

[0007] The image processing system disclosed herein performs multiple image searches to explore images containing features of a model image from object images, i.e., extracting images. K is set as a natural number variable. N and M are set as natural number constants, where M is greater than N. In the first image search, a first candidate image is explored from the object images with a first compression ratio, and the first candidate image is added to the candidates for the extracted images. The first candidate image contains features of the model image with the first compression ratio. In the (k+1)th image search, preliminary candidates for the (k+1)th image search are selected from the object images with a compression ratio smaller than the kth compression ratio. The preliminary candidates for the (k+1)th image search include the candidates for the extracted images obtained through the kth image search. Further, in the (k+1)th image search, a (k+1)th candidate image is explored from the preliminary candidates used in the (k+1)th image search, and the (k+1)th candidate image is added to the new candidates for the extracted image. The (k+1)th candidate image contains features of the model image with the (k+1)th compression ratio. The image processing system includes a search processing unit, an acquisition unit, and a sub-exploration unit. The search processing unit performs multiple image searches. The acquisition unit acquires relevant information indicating the correlation between the Nth candidate image and the sub-candidate image. The sub-exploration unit acquires the sub-candidate image based on the Nth candidate image obtained through the Nth image search and the relevant information acquired by the acquisition unit, and adds the sub-candidate image to the preliminary candidates used in the Mth image search.

[0008] This disclosure has the advantage of enabling high-speed processing. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an image processing method according to one embodiment.

[0010] Figure 2 It is a block diagram of an image processing system that implements the above image processing methods.

[0011] Figure 3 This is the model image with the first compression ratio used in the first example of the image processing method described above.

[0012] Figure 4 It is the object image with the first compression rate used in the first example of the image processing method described above.

[0013] Figure 5 This is the model image with the fourth compression rate used in the first example of the image processing method described above.

[0014] Figure 6AThis is the object image with the fourth compression rate used in the first example of the image processing method described above.

[0015] Figure 6B This is the object image with the fourth compression rate used in the first example of the image processing method described above.

[0016] Figure 7 This is the model image used in the second example of the image processing method described above.

[0017] Figure 8 This is the object image used in the second example of the image processing method described above.

[0018] -Symbol Explanation-

[0019] 51 Acquisition Department

[0020] 52 Sub-Exploration Department

[0021] S1 Search Processing Department

[0022] X1 image processing system. Detailed Implementation

[0023] The image processing method, program, and image processing system X1 according to the embodiments will be described below using the accompanying drawings. However, the embodiments described below are merely one of the various embodiments of this disclosure. Various modifications can be made according to design, etc., to achieve the objectives of this disclosure. Furthermore, the figures described in the embodiments below are schematic diagrams, and the ratios of the size and thickness of the structural elements in the figures may not reflect the actual size ratios.

[0024] (1) Summary

[0025] In image processing methods, image searching is performed. Image searching is the process of comparing an image of the object being inspected, generated by photographing the object, with a pre-prepared model image. By performing image searching, the shape, position, and orientation of the object being inspected are determined. As an example, image processing methods are performed in the manufacturing process of components (inspection objects) such as products in a factory for the purpose of visual inspection of the components. As another example, image processing methods are performed to identify a component from other components.

[0026] Furthermore, the image search is performed multiple times. Let F be the number of image searches (F is a constant natural number). In at least a portion of these multiple image searches, compressed images of both the object image and the model image are used to accelerate processing. Each time an image search is repeated, the compression ratio of both the object image and the model image is reduced. In other words, each time an image search is repeated, the resolution of both the object image and the model image is increased.

[0027] Through the first image search, the candidate images containing features of the model image—that is, the positions and orientations of the extracted images—are filtered to some extent. The extracted image is the object finally determined through multiple (F) image searches. In other words, the extracted image is obtained through the Fth (final) image search. In the (k+1)th image search (k is a natural number variable, i.e., k = 1, 2, 3, ..., F), further filtering is performed, limited to the periphery of the position selected through the kth image search, and orientations close to those selected through the kth image search. Thus, compared to performing an image search across the entire uncompressed (high-resolution) object image, the processing time required to determine the extracted image can be shortened.

[0028] Figure 1 This is a flowchart illustrating the image processing method of this embodiment. The image processing method of this embodiment performs multiple image searches to explore images containing features of the model image from object images, i.e., extracting images. k is set as a natural number variable. N and M are natural number constants, and M is greater than N. In the first image search (step ST4), a first candidate image is explored from object images with a first compression ratio, and the first candidate image is added to the candidates for the extracted image. The first candidate image contains features of the model image with the first compression ratio. In the (k+1)th image search, preliminary candidates for the (k+1)th image search are selected from object images with a compression ratio smaller than the kth compression ratio. The preliminary candidates for the (k+1)th image search include candidates for the extracted image obtained through the kth image search. Further, in the (k+1)th image search, a (k+1)th candidate image is explored from the preliminary candidates for the (k+1)th image search, and the (k+1)th candidate image is added to the new candidates for the extracted image. The (k+1)th candidate image contains features of the model image with the (k+1)th compression ratio. The image processing method includes a search process, an acquisition process, and a sub-exploration process. The search process performs multiple image searches. The acquisition process acquires relevant information representing the relationship between the Nth candidate image and the sub-candidate images (step ST1). The sub-exploration process, based on the Nth candidate image obtained through the Nth image search and the relevant information acquired through the acquisition process, acquires sub-candidate images and adds them to the preliminary candidates for the Mth image search (step ST6). The relevant information includes, for example, information related to the rotation angle difference between the Nth candidate image and the sub-candidate images. In this case, a sub-candidate image refers to an image in which the Nth candidate image is rotated by the aforementioned rotation angle difference.

[0029] In summary, the k-th candidate image (the main candidate) is obtained through the k-th image search, and this candidate image is added to the candidate pool for the (k+1)-th image search. When k+1 = M, the sub-candidate image is also added to the candidate pool for the (k+1)-th image search. In the (k+1)-th image search, the (k+1)-th candidate image is obtained from the candidate pool used in the (k+1)-th image search. If the k-th image search is the final image search (in other words, when k = F), the k-th candidate image is output as the final search result (extracted image).

[0030] According to this embodiment, compared to performing image search across the entire uncompressed object image, the processing time required to determine the extracted image can be shortened.

[0031] Furthermore, in the Nth image search, due to the high compression ratio of the object image and the model image, data corresponding to the differences between the uncompressed object image and the uncompressed model image may be lacking. Therefore, in the uncompressed state, an image inconsistent with the model image may be obtained as the Nth candidate image. In this case, the Nth candidate image becomes one of the preliminary candidates (the search range of the N+1th image search) for the N+1th image search, and the Nth candidate image is determined to be inconsistent with the model image through subsequent image searches. However, in the Mth image search after the Nth image search, not only the candidate images obtained from previous image searches, but also the sub-candidate images are used as preliminary candidates for the Mth image search, so the sub-candidate images can be determined as the extracted images. In other words, even if the Nth candidate image is inconsistent with the model image in the uncompressed state, it is possible that the sub-candidate image is consistent with the model image in the uncompressed state. In the image processing method of this embodiment, compared to the case where no sub-candidate image is added to the pre-candidate image, the time required until the sub-candidate image is found can be shortened, and thus the time required until the image is determined to be extracted can be shortened.

[0032] Furthermore, N=1 is preferred. In this embodiment, it is stated that N=1. In other words, a sub-candidate image is obtained based on the first candidate image obtained through the first image search and related information. The sub-candidate image is added to the reserve candidates used in the Mth image search. In the Mth image search, a new candidate image (the Mth candidate image) is explored and extracted from the (M-1)th candidate image obtained through the (M-1)th image search and the reserve candidates. In this embodiment, it is stated that M=4. Moreover, in this embodiment, the Mth image search is the last image search among multiple image searches. In other words, in this embodiment, F=M.

[0033] Figure 1 The flowchart shown is just one example of an image processing method. The order of processing can be changed, and processing steps can be added or omitted as appropriate.

[0034] The image processing method is executed through the image processing system X1. For example... Figure 2 As shown, the image processing system X1 includes a search processing unit S1, an acquisition unit 51, and a sub-exploration unit 52. The search processing unit S1 performs multiple image searches. The acquisition unit 51 acquires relevant information indicating the relationship between the Nth candidate image and the sub-candidate image. The sub-exploration unit 52, based on the Nth candidate image obtained through the Nth image search and the relevant information acquired by the acquisition unit 51, obtains the sub-candidate image and adds it to the reserve candidates for the Mth image search.

[0035] By using the image processing system X1, the processing time required until the image extraction is determined can be reduced.

[0036] (2) Image processing system

[0037] The image processing system X1 includes a computer system having one or more processors and memory. By executing a program recorded in the computer system's memory through the computer system's processor, at least a portion of the functions of the image processing system X1 can be implemented. The program can be recorded in memory, provided via electrical communication lines such as the Internet, or provided via a non-transitory recording medium such as a memory card.

[0038] The image processing system X1 includes a search processing unit S1, an acquisition unit 51, a sub-exploration unit 52, a generation processing unit 53, a storage unit 54, a control unit 55, and an image processing unit 56. The search processing unit S1 includes a first search unit 1, a second search unit 2, a third search unit 3, and a fourth search unit 4. Furthermore, these only represent the functions implemented by the image processing system X1 and do not necessarily represent any specific structure of an entity.

[0039] exist Figure 2 In the image processing system X1, an image processing unit 61, a drive mechanism 62, and an operation unit 63 are provided as external structures. Alternatively, at least one of the image processing unit 61, the drive mechanism 62, and the operation unit 63 may also be part of the image processing system X1.

[0040] (3) Camera Department

[0041] The camera unit 61 is, for example, a two-dimensional image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The camera unit 61 captures an image of the object (component) being inspected. Thus, the camera unit 61 generates an object image reflecting the image of the object being inspected. The object being inspected is transported into the imaging range of the camera unit 61 by a moving device, such as a belt conveyor. The timing of the image capture by the camera unit 61 is controlled by the control unit 55.

[0042] (4) Drive mechanism

[0043] The drive mechanism 62 includes, for example, a robotic arm or an air generating device. The drive mechanism 62 moves the object being inspected. For example, if an object is determined to be a qualified product by the image processing system X1, the robotic arm holds (picks up) the object and moves it to the location for the next process. Conversely, if an object is determined to be a non-qualified product by the image processing system X1, the air generating device removes the object using air. The operation of the drive mechanism 62 is controlled by the control unit 55.

[0044] (5) Storage section

[0045] The storage unit 54 is, for example, ROM (Read Only Memory), RAM (Random Access Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 54 stores the model image. Furthermore, the storage unit 54 stores various parameters used in the image processing method.

[0046] (6) Image Processing Department

[0047] The image processing unit 56 compresses (compression processing) the object image generated by the camera unit 61. Furthermore, the image processing unit 56 acquires a model image from the storage unit 54 and compresses (compression processing) the model image. More specifically, the image processing unit 56 transforms both the object image and the model image into images with a smaller pixel count (compressed images).

[0048] Both the object image and the model image are two-dimensional images. Furthermore, both the object image and the model image are digital images. In this embodiment, we will assume that both the object image and the model image are black and white images. However, the object image and the model image may also be grayscale images or color images.

[0049] The image processing unit 56 generates, for example, a 1 / 2i compressed image where the pixel count of the original image (object image and model image) is set to 1 / 2i times (i is a variable of natural numbers). In this embodiment, the image processing unit 56 generates 1 / 2 compressed images, 1 / 4 compressed images, and 1 / 8 compressed images.

[0050] Furthermore, the image processing unit 56 extracts contour lines from the compressed and uncompressed images of the object image and the model image, respectively. Algorithms for extracting contour lines can be, for example, the Sobel method or the Laplacian method of Gauss. Alternatively, the model image can be provided from the outset as an image consisting only of contour lines.

[0051] (7) Operation Section

[0052] The operation unit 63 is a user interface that accepts user operations. The operation unit 63 includes, for example, at least one of a button, a switch, a dual in-line switch, a touch panel, and a touch panel display.

[0053] The operation unit 63 accepts user operations for determining relevant information.

[0054] In addition, the operation unit 63 accepts user operations for determining the setting angle (the rotation angle that rotates the kth candidate image or sub-candidate image) described later.

[0055] Furthermore, the operation unit 63 accepts user input to determine the number of image searches.

[0056] Furthermore, the operation unit 63 only needs to accept operations from the user; the entity that actually operates the operation unit 63 is not limited to the user. The operation unit 63 can also accept operations from people other than the user.

[0057] (8) Acquisition Department

[0058] The acquisition unit 51 acquires relevant information from the operation unit 63. That is, if the operation unit 63 performs an operation to determine relevant information, the operation unit 63 outputs a signal corresponding to the content of the operation, and the acquisition unit 51 acquires the signal as relevant information.

[0059] In other words, the image processing method of this embodiment includes relevant information determination processing performed by the acquisition unit 51, which determines relevant information based on the user's operation.

[0060] Furthermore, the acquisition unit 51 acquires information related to the set angle from the operation unit 63. That is, if the operation unit 63 is used to determine the set angle, the operation unit 63 outputs a signal corresponding to the operation content, and the acquisition unit 51 acquires the signal as information related to the set angle.

[0061] In other words, the image processing method of this embodiment includes a setting process performed by the acquisition unit 51, which is a process of acquiring information related to a setting angle. Furthermore, the image processing method of this embodiment includes a setting angle determination process performed by the acquisition unit 51, which is a process of determining the setting angle based on user operations. The setting angle is, for example, the angle specified by the user through operation of the operation unit 63.

[0062] (9) Search Processing Department

[0063] The search processing unit S1 includes multiple search units. In this embodiment, there are four search units. That is, the search processing unit S1 comprises multiple search units, including a first search unit 1, a second search unit 2, a third search unit 3, and a fourth search unit 4.

[0064] Multiple search units perform image searches separately. An image search is a process of comparing an object image generated from photographing the object to be inspected with a pre-prepared model image. For a single model image, the multiple search units perform image searches sequentially. Search unit 1, search unit 2, search unit 3, and search unit 4 perform image searches in sequence. The number of image searches is preferably three or more. In other words, in the search process (the process of performing multiple image searches), it is preferable to perform three or more image searches. In this embodiment, the number of image searches is four, but it is not limited to four; two or more searches are sufficient.

[0065] Multiple search units perform image searches using object images and model images with different compression ratios. In the image processing method of this embodiment, the compression ratio of the object image and model image is reduced each time an image search is repeated. That is, the first search unit 1 performs a first image search using object images and model images with a first compression ratio, the second search unit 2 performs a second image search using object images and model images with a second compression ratio, the third search unit 3 performs a third image search using object images and model images with a third compression ratio, and the fourth search unit 4 performs a fourth image search using object images and model images with a fourth compression ratio. The (k+1)th compression ratio is a compression ratio smaller than the kth compression ratio. For example, in the first image search, a 1 / 8 compressed image is used; in the second image search, a 1 / 4 compressed image is used; in the third image search, a 1 / 2 compressed image is used; and in the fourth (final) image search, an uncompressed image is used. In other words, the compression ratio may also include 0 (uncompressed).

[0066] (10) Sub-exploration Department

[0067] The sub-exploration unit 52 obtains a sub-candidate image based on the first (Nth) candidate image obtained through the first (Nth) image search and the relevant information obtained by the acquisition unit 51. As an example, the relevant information includes information related to the rotation angle difference between the first (Nth) candidate image and the sub-candidate image. In this case, the sub-exploration unit 52 generates an image that rotates the first (Nth) candidate image by the aforementioned rotation angle difference, and sets the generated image as the sub-candidate image. The information related to the aforementioned rotation angle difference is generated, for example, by the generation processing unit 53. The information related to the aforementioned rotation angle difference can also be provided through user operation of the operation unit 63. For example, if only one type of component is set as the inspection object, it is not necessary to change the information related to the aforementioned rotation angle difference later, so the user can provide the information related to the aforementioned rotation angle difference in advance.

[0068] As the aforementioned rotation angle difference, only one angle difference can be used, or multiple angle differences can be used. For example, as the aforementioned rotation angle difference, three angle differences of 90 degrees, 180 degrees, and 270 degrees can also be used. In this case, the sub-exploration unit 52 generates three sub-candidate images that rotate the first candidate image by 90 degrees, 180 degrees, and 270 degrees, respectively.

[0069] (11) Generation Processing Unit

[0070] The image processing method of this embodiment includes a generation process performed by the generation processing unit 53, which generates information related to the rotation angle difference based on the rotational symmetry of the subject in the model image. For example, if the subject in the model image is L-fold rotationally symmetric (L is a constant natural number), the generation processing unit 53 determines the rotation angle difference to be 360 / L [degrees].

[0071] For example, Figure 3 The subject (object to be inspected, Ob1) of the model image shown is rotationally symmetric twice, so the generation processing unit 53 determines the above rotation angle difference to be 180 degrees.

[0072] Information related to the rotational symmetry of the subject in the model image can also be provided through user operation of the operation unit 63. Alternatively, the generation processing unit 53 can generate information related to the rotational symmetry of the subject in the model image by parsing the model image.

[0073] For example, the timing of transporting an inspection object Ob1 into the camera range of the camera unit 61, and the timing of transporting an inspection object of a different shape into the camera range of the camera unit 61, are predetermined. Therefore, the generation processing unit 53 changes the information related to the aforementioned rotation angle difference according to the time.

[0074] Furthermore, when the subject is not rotationally symmetric, the generation processing unit 53 outputs information indicating this meaning.

[0075] (12)Example 1

[0076] Reference Figure 1 , Figures 3 to 6B This is the first example illustrating the process of inspecting an object using image processing methods.

[0077] First, the user operates the operation unit 63 to input relevant information. The acquisition unit 51 acquires the relevant information from the operation unit 63 (step ST1). The camera unit 61 captures an image of the inspection object Opl to generate an object image (step ST2).

[0078] Figure 3 The image represents the model with the first compression ratio. Figure 4 This represents the object image with the first compression ratio. Figure 5 This represents the model image at the 4th compression ratio (uncompressed). Figure 6A , Figure 6B This represents the object image at the 4th compression ratio (uncompressed). However, in Figures 3 to 6B In both cases, the model image and the object image are images consisting only of outlines (shaded areas in the figure). The image processing unit 56 compresses the object image as needed and generates the object image by extracting the outlines (step ST3). Furthermore, the model image is a rectangular or square image, and rows and columns that do not contain outlines are removed beforehand.

[0079] In addition, the resolution ratio of the preferred model image and the object image is... Figures 3 to 6B The resolution shown is high, but for the sake of simplicity, in Figures 3 to 6B In the image, the image is represented by a lower resolution.

[0080] like Figure 5 As shown, the two-dimensional shape of the inspection object Ob1 is cross-shaped. The inspection object Ob1 has a central portion Ob10 and four protrusions Ob11 to Ob14. The four protrusions Ob11 to Ob14 protrude from the central portion Ob10. Protrusions Ob11 and Ob12 are positioned on opposite sides of the central portion Ob10. Protrusions Ob13 and Ob14 are also positioned on opposite sides of the central portion Ob10. The protruding directions of protrusions Ob11 and Ob12 are orthogonal to the protruding directions of protrusions Ob13 and Ob14.

[0081] exist Figure 5 In the middle, the lengths of the four protrusions Ob11 to Ob14 are 8 [pixels], 9 [pixels], 6 [pixels], and 6 [pixels], respectively. If... Figure 5 If the model image is compressed to the first compression ratio, then as follows: Figure 3 As shown, the lengths of the four protrusions Ob11 to Ob14 are 4 pixels, 4 pixels, 3 pixels, and 3 pixels, respectively. In other words, in Figure 3 In the middle, the protrusions Ob11 and Ob12 are of equal length. Figure 3 In the analysis, the two-dimensional shape of the object Ob1 is found to be rotationally symmetric twice.

[0082] Figure 6B It is relative to Figure 6A The image shows the object being inspected, Opl, rotated 180 degrees. Figure 6A , Figure 6B The case where any object image is compressed to the first compression ratio is also... Figure 4 The image shown. In other words, in Figure 3 , Figure 4 In the model image and object image shown with the first compression ratio, there is no distinction between the state in which the protrusion Obll is on the paper and the state in which it is below the paper relative to the center Ob10.

[0083] In image search, the model image is compared with the object image by scanning the object image.

[0084] (12-1) First image search

[0085] In the first image search (step ST4), the first search unit 1 pairs... Figure 3 Model images and Figure 4 The image is compared with the object image. First, the first search unit 1 designates the upper left corner of the object image as the comparison object (comparison region R1). The size of the comparison region R1 is equal to the size of the model image. The first search unit 1 compares each pixel of the image in the comparison region R1 with each pixel of the model image. If the similarity rate between the image in the comparison region R1 and the model image is above a threshold, the first search unit 1 determines that the image in the comparison region R1 is a first candidate image containing features of the model image. In other words, in this case, the first search unit 1 adds the image in the comparison region R1 to the candidates for the extracted image. On the other hand, if the similarity rate between the image in the comparison region R1 and the model image is less than a threshold, the first search unit 1 determines that the image in the comparison region R1 is not a first candidate image.

[0086] The threshold is pre-stored in the storage unit 54. In this embodiment, it is assumed that the threshold is 100%. In other words, in this embodiment, the first search unit 1 determines that the image of the comparison region R1 is the first candidate image only if the image of the comparison region R1 is completely consistent with the model image.

[0087] The first search unit 1 shifts the comparison region R1 by 1 [pixel] increments, and each time it shifts by 1 [pixel], it compares the image of the comparison region R1 with the model image. For example, it shifts the comparison region R1 in the column direction ( Figure 4 The comparison region R1 is shifted 1 pixel at a time (left-right direction) on the paper. If the comparison region R1 reaches one end (right or left) of the object image, it is shifted 1 pixel in the row direction. This process is considered as one set. By repeating this process multiple times, the first search unit 1 compares the entire area of ​​the object image with the model image. Figure 4 In the process, when region Rll is selected as comparison region R1, the first search unit 1 determines that the image in comparison region R1 is the first candidate image.

[0088] Furthermore, the first search unit 1 compares multiple rotated images with the object image. These multiple rotated images are images in which the model image is rotated in stages. For example, the model image is rotated 360 degrees clockwise or counterclockwise in 1-degree increments to generate multiple rotated images. By repeating the above process multiple times for each rotated image, the first search unit 1 compares the entire area of ​​the object image with the rotated images to explore first candidate images.

[0089] Here, if a first (Nth) candidate image is found at a certain location, the rotated image at that location can also be removed from the search range of the first (Nth) image search. In other words, the rotated image at that location may not be selected as the first (Nth) candidate image. This reduces the time required for image searching. The rotated image at that location is compared with the model image in the fourth image search to determine if it corresponds to the extracted image.

[0090] If at least one first candidate image is found through the first image search, the storage unit 54 stores information related to the first candidate image. The information related to the first candidate image includes at least the position and orientation of the first candidate image. The information related to the first candidate image may also include the consistency rate between the first candidate image and the model image.

[0091] As mentioned above, in Figure 3 , Figure 4 In the model image and object image shown with the first compression ratio, there is no distinction between the state where the protrusion Ob11 is on the upper side of the paper and the state where it is on the lower side of the paper relative to the center Ob10. In any state, when region R11 is selected as comparison region R1, the first search unit 1 determines that the image of comparison region R1 is the first candidate image.

[0092] If, for at least one region, the image in that region is determined to be the first candidate image (step ST5: Yes), then the sub-exploration unit 52 generates a sub-candidate image (step ST6: sub-exploration processing). Specifically, the sub-exploration unit 52 sets the image in which the first candidate image is rotated by 180 degrees as the sub-candidate image.

[0093] (12-2) Image search after the second time

[0094] Next, the second search unit 2 performs a second image search using the object image with the second compression ratio and the model image with the second compression ratio (step ST7). In the second image search, firstly, preliminary candidates for the second image search are selected. The preliminary candidates for the second image search include the first candidate image obtained through the first image search. In this example, through the first image search, region R11 (refer to...) Figure 4 The image corresponding to the first candidate image is obtained. The storage unit 54 stores information about the position and orientation of the first candidate image. Therefore, using this information, the second search unit 2 determines the region corresponding to region R11 in the object image of the second compression ratio, and sets the image of the region corresponding to region R11 as a preliminary candidate for the second image search. The second search unit 2 then searches for a second candidate image from the preliminary candidates for the second image search, exploring for features of the model image containing the second compression ratio.

[0095] Similarly, in the third and fourth image searches (steps ST9 and ST11), candidate images are explored from the images of the regions obtained through previous image searches (preliminary candidates). In other words, in the (k+1)th image search (here, k = 1, 2, 3), preliminary candidates (the search range of the (k+1)th image search) are selected from the object images of the (k+1)th compression ratio. Furthermore, from the preliminary candidates used in the (k+1)th image search, a (k+1)th candidate image containing features of the model image of the (k+1)th compression ratio is explored.

[0096] The candidate images used in the (k+1)th image search include not only the candidate images obtained from the kth image search (the kth candidate image), but also the surrounding images of the candidate images obtained from the kth image search. For example, Figure 6A In this process, the region corresponding to the candidate image obtained through the third image search is designated as region R41. The preliminary candidate image used in the fourth image search includes the image of region R41 and the surrounding images of region R41. In the fourth image search, a fourth candidate image is explored from the image of region R41 and the surrounding images of region R41.

[0097] The so-called image surrounding region R41 refers to the image of the region that has been moved by a predetermined amount relative to region R41. More specifically, the so-called image surrounding region R41 refers to the image of the region formed by at least one of moving region R41 in parallel by an amount less than a predetermined number of pixels or rotating region R41 by a rotation angle less than a predetermined rotation angle.

[0098] When k = 1 or 2, the candidate images used in the (k+1)th image search only include the candidate images obtained from the kth image search and their surrounding images. When k = 3, the candidate images used in the (k+1)th (Mth)th image search include not only the candidate images obtained from the kth image search and their surrounding images, but also sub-candidate images. In this example, the sub-candidate image is the image in which the first candidate image is rotated 180 degrees. In other words, the candidate images used in the 4th image search at least include... Figure 6A The image of region R41, the image of region R41 rotated 180 degrees, i.e. Figure 6B The image of region R42.

[0099] Furthermore, when k=3, the candidate images used for the (k+1)th (Mth) image search also include multiple rotated images. These multiple rotated images are images in which the kth candidate image or a sub-candidate image is rotated in stages within a range from 0 degrees to a set angle (a first set value). In other words, the image processing method of this embodiment includes a range expansion process, which is the process of adding multiple rotated images to the candidate images used for the (k+1)th image search. The range expansion process is performed, for example, by the fourth search unit 4. Multiple rotated images are generated based on the kth candidate image and the sub-candidate images, respectively.

[0100] For example, multiple rotating images are generated by rotating the third candidate image (k-th candidate image) or a sub-candidate image clockwise and counterclockwise at a second set value (a value representing an angle). Furthermore, an upper limit, i.e., a first set value, is predetermined for the difference in rotation angle between the third candidate image or sub-candidate image and the rotating images. The first and second set values ​​are determined based on the operation performed on the operation unit 63. That is, if the operation unit 63 is operated to determine at least one of the first and second set values, the operation unit 63 outputs a signal corresponding to the content of the operation, and based on this signal, the fourth search unit 4 determines the first and second set values.

[0101] The first setting value is smaller than half the difference in rotation angle between the first (Nth) candidate image and the sub-candidate image. The first setting value can be selected from multiple values, for example, 5 degrees, 15 degrees, and 45 degrees.

[0102] The second setting is smaller than the first setting. For example, the second setting is 1 degree.

[0103] In the final (4th) image search, if a (4th) candidate image is found (step ST12: Yes), the search processing unit S1 sets this candidate image as the final search result. In other words, the search processing unit S1 sets this candidate image as an image containing the features of the model image, i.e., an extracted image. The search processing unit S1 outputs information related to the extracted image. For example, the search processing unit S1 outputs information related to the extracted image to the control unit 55. The information related to the extracted image includes, for example, information about the position and orientation of the extracted image. Based on the information related to the extracted image, the position and orientation of the object being photographed, i.e., the object being inspected, Ob1, can be determined. The information related to the extracted image may also include the consistency rate between the extracted image and the model image.

[0104] The control unit 55 controls the drive mechanism 62 based on information related to the extracted image. The drive mechanism 62, for example, holds (picks up) the inspection object Ob1 and moves it to the location for the next process (step ST13). Furthermore, the drive mechanism 62 corrects the orientation of the inspection object Ob1 based on information indicating its orientation.

[0105] Furthermore, if no candidate image is found in the image searches between the first and last (fourth) image searches (the second and third searches) (step ST8 or ST10: no), at least the sub-candidate image is included in the preliminary candidates for the fourth image search, and the fourth image search is performed (step ST14). In other words, the sub-candidate image is compared with the model image to explore for a fourth candidate image. Here, in addition to the sub-candidate image, a rotated image that rotates the sub-candidate image may also be included in the preliminary candidates for the fourth image search.

[0106] (12-3) Advantages

[0107] In the first example, in the second and third image searches, only the candidate images for extraction obtained from previous image searches and their surrounding images are included in the search scope (preliminary candidates). Therefore, compared to including the entire object image in the search scope, the time required for the second and third image searches can be shortened. Furthermore, in the second and third image searches, only the candidate images for extraction obtained from previous image searches can be included in the search scope, and the surrounding images of the candidate images can be removed from the search scope.

[0108] Furthermore, in the first example, during the fourth image search, a sub-candidate image was also added to the search scope (preliminary candidate). Since the sub-candidate image was not obtained directly through image search but rather derived from the first candidate image, the time required to find the sub-candidate image can be shortened. Therefore, when the sub-candidate image is output as the final search result (extracted image), the time required to determine the extracted image can be shortened.

[0109] Furthermore, the image processing method of this embodiment explores and extracts images based on a single model image. In other words, a single model image is transformed to generate multiple rotated images. Therefore, compared to preparing multiple rotated images in advance, the storage capacity occupied by the model image can be reduced.

[0110] (13)Example 2

[0111] Reference Figure 7 , Figure 8 This is the second example illustrating the process of inspecting an object using image processing methods. Figure 7 A model image representing a certain compression ratio. Figure 8 An object image representing a certain compression ratio.

[0112] In the second example, the two-dimensional shape of the object Ob2 is that of a gear. The object Ob2 has a disk-shaped portion Ob20 and multiple ( Figure 7 There are 18 teeth (Ob21) in the middle. Multiple teeth Ob21 are arranged at equal intervals on the outer edge of the disk portion Ob20. The length of one of the multiple teeth Ob21, Ob210, is shorter than the other teeth Ob21.

[0113] exist Figure 8 In this context, region R8 is the region corresponding to the entire object image. The image that is correctly extracted through image search is the image in region R81.

[0114] However, in the first image search, the object image and the model image using the first compression ratio have a high compression ratio. Therefore, the shape difference between the shorter tooth Ob210 and the other teeth Ob21 among the multiple teeth Ob21 may become smaller. As a result, in the first image search, the image of region R82 may be output as a candidate image for extraction (the first candidate image). The image of region R82 is an image rotated by a specified angle relative to the image of region R81.

[0115] Therefore, the sub-exploration unit 52 generates multiple images that rotate the first candidate image (the image of region R81), and sets each of the generated images as a sub-candidate image. The multiple sub-candidate images are, for example, images that rotate the first candidate image by a 360 / NA [degree] scale. NA is the number of teeth Ob21. In other words, the multiple sub-candidate images are images that rotate the first candidate image by 20 degrees, 40 degrees, 60 degrees, ..., 340 degrees.

[0116] Here, the number NA of teeth Ob21 is information related to the rotational symmetry of the subject (object of inspection Ob2) in the model image. The information regarding the number NA of teeth Ob21 can also be provided by the user's operation of the operation unit 63. Alternatively, the generation processing unit 53 can also obtain the number NA of teeth Ob21 by analyzing the model image with the first (Nth) compression ratio.

[0117] One of the multiple candidate images is the image of region R81. Therefore, in the fourth image search that includes multiple candidate images in the search range (preliminary candidates), the image of region R81 is determined to be the extracted image.

[0118] In the second example, similarly to the first, the time required for image search was reduced.

[0119] (Modifications of the implementation method)

[0120] The following are examples of variations of the implementation method. These variations can also be implemented through appropriate combinations.

[0121] In this implementation, the sub-candidate image is added to the candidate pool only in the last (4th) image search out of multiple (F) image searches. Alternatively, the sub-candidate image may be added to the candidate pool in the Nth (N=1) image search and before the last image search (the 2nd or 3rd). Furthermore, the sub-candidate image may be added to the candidate pool through two or more image searches out of multiple image searches. For example, the sub-candidate image may be added to the candidate pool in all of the image searches following the Nth (N=1) image search (the 2nd to 4th).

[0122] The images used in image search are not limited to those from which contour lines are extracted. The images used in image search can also be raw data generated by the camera unit 61, or data that has been appropriately transformed from the raw data.

[0123] The extracted image is obtained through image processing methods. However, the final output information of the image processing method does not necessarily have to be the extracted image; for example, it can include the position (coordinates) and orientation information of the extracted image.

[0124] When the subject of the model image is not rotationally symmetric, the processing of generating sub-candidate images can also be invalidated.

[0125] Image processing methods can also be realized through (computer) programs or non-transitory recording media containing programs. That is, the program involved in one approach is a program for causing one or more processors to execute the image processing method.

[0126] The image processing system X1 of this disclosure includes a computer system. The computer system is primarily structured around a processor and memory as hardware. By executing a program recorded in the computer system's memory, the processor can realize the functions of the image processing system X1 of this disclosure. The program can be pre-recorded in the computer system's memory, provided via electrical communication lines, or provided via a non-transitory recording medium such as a memory card, optical disc, or hard disk drive that can be read by the computer system. The processor of the computer system includes one or more electronic circuits, which include semiconductor integrated circuits (ICs) or large-scale integrated circuits (LSIs). The term ICs or LSIs used here is used depending on the degree of integration, including integrated circuits referred to as system LSIs, VLSIs (Very Large Scale Integration), or ULSIs (Ultra Large Scale Integration). Furthermore, a logic device capable of reconstructing the internal bonding relationships of an FPGA (Field-Programmable Gate Array) or an LSI, or reconstructing the internal circuit partitioning of an LSI, can also be used as a processor. Multiple electronic circuits can be aggregated onto a single chip or distributed across multiple chips. Multiple chips can be integrated into a single device or distributed across multiple devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Therefore, a microcontroller also includes one or more electronic circuits, which may comprise semiconductor integrated circuits or large-scale integrated circuits.

[0127] Furthermore, the integration of multiple functions in the image processing system X1 into a single device is not a necessary structure for the image processing system X1, and the structural elements of the image processing system X1 can be distributed across multiple devices. Moreover, at least some of the functions of the image processing system X1, such as at least some of the functions of the search processing unit S1, can also be implemented via the cloud (cloud computing) or the like.

[0128] Conversely, in the embodiments, at least a portion of the functions of the image processing system X1, etc., which are distributed across multiple devices, can be integrated into one device. For example, the functions distributed across the image processing system X1 and the drive mechanism 62 can also be integrated into one device.

[0129] (Summarize)

[0130] Based on the implementation methods described above, the following methods are disclosed.

[0131] The image processing method involved in the first approach involves multiple image searches to explore images containing features of the model image from the object images, i.e., extracting images. k is a natural number variable. N and M are natural number constants, with M being larger than N. In the first image search, a first candidate image is explored from the object images with a first compression ratio, and this first candidate image is added to the candidates for the extracted images. The first candidate image contains features of the model image with the first compression ratio. In the (k+1)th image search, preliminary candidates for the (k+1)th image search are selected from the object images with a compression ratio smaller than the kth compression ratio. The preliminary candidates for the (k+1)th image search include the candidates for the extracted images obtained through the kth image search. Further, in the (k+1)th image search, a (k+1)th candidate image is explored from the preliminary candidates for the (k+1)th image search, and this (k+1)th candidate image is added to the new candidates for the extracted images. The (k+1)th candidate image contains features of the model image with the (k+1)th compression ratio. The image processing method includes search processing, acquisition processing, and sub-exploration processing. The search processing performs multiple image searches. The acquisition processing acquires relevant information representing the relationship between the Nth candidate image and the sub-candidate images. The sub-exploration processing, based on the Nth candidate image obtained through the Nth image search and the relevant information acquired through the acquisition processing, obtains sub-candidate images and adds them to the preliminary candidates for the Mth image search.

[0132] The above structure enables high-speed processing for determining the extracted image.

[0133] Furthermore, in the image processing method involved in the second approach, in the first approach, the relevant information includes information related to the rotation angle difference between the Nth candidate image and the sub-candidate image.

[0134] The above structure enables the high-speed extraction of images of subjects containing rotational symmetry.

[0135] Furthermore, the image processing method involved in the third approach, compared to the second approach, also includes a generation process. In the generation process, information related to the rotational angle difference is generated based on the rotational symmetry of the subject in the model image.

[0136] The above structure reduces the hassle of inputting rotation angle differences.

[0137] Furthermore, the image processing method involved in the fourth method, in any of the methods 1 to 3, also includes relevant information determination processing. In the relevant information determination processing, relevant information is determined based on the user's operation.

[0138] The above structure allows for the modification of relevant information based on the situation.

[0139] Furthermore, the image processing method involved in the fifth method, in any of the methods 1 to 4, also includes range expansion processing and setting processing. In the range expansion processing, multiple rotated images are added to the pool of candidates for the (k+1)th image search. The multiple rotated images are images that rotate the k-th candidate image or a sub-candidate image in stages within a range from 0 degrees to a set angle. In the setting processing, information related to the set angle is obtained.

[0140] The above structure can improve the accuracy of image search.

[0141] Furthermore, the image processing method involved in the sixth method also includes an angle setting process in the fifth method. In the angle setting process, the setting angle is determined based on the user's operation.

[0142] The above structure allows for adjustments to the set angle based on the situation.

[0143] Furthermore, in the image processing method involved in the seventh method, N = 1 in any of the methods from the first to the sixth.

[0144] With the above structure, as a result of the first image search, the first candidate image and sub-candidate images are added to the preliminary candidates. Therefore, the possibility of insufficient preliminary candidates in the first image search can be reduced, and the accuracy of the image search can be improved.

[0145] Furthermore, in the image processing method involved in the eighth method, in any one of the methods from the first to the seventh method, the image search is performed more than three times during the search process.

[0146] The above structure can improve the accuracy of image search.

[0147] Furthermore, the image processing method involved in the 9th method, in any of the 1st to 8th methods, explores and extracts images based on a model image.

[0148] The above structure can reduce the storage capacity occupied by the model image.

[0149] For structures other than the first method, which are not essential for image processing methods, they can be appropriately omitted.

[0150] Furthermore, the program involved in the 10th method is a program for causing one or more processors to execute any of the image processing methods involved in the 1st to 9th methods.

[0151] The above structure enables high-speed processing for determining the extracted image.

[0152] Furthermore, the image processing system (X1) involved in method 11 performs multiple image searches to explore images containing features of the model image from the object images, i.e., extracting images. k is a natural number variable. N and M are natural number constants, with M being larger than N. In the first image search, a first candidate image is explored from the object images with a first compression ratio, and this first candidate image is added to the candidates for the extracted images. The first candidate image contains features of the model image with the first compression ratio. In the (k+1)th image search, preliminary candidates for the (k+1)th image search are selected from the object images with a compression ratio smaller than the kth compression ratio. The preliminary candidates for the (k+1)th image search include the candidates for the extracted images obtained through the kth image search. Further, in the (k+1)th image search, a (k+1)th candidate image is explored from the preliminary candidates for the (k+1)th image search, and this (k+1)th candidate image is added to the new candidates for the extracted images. The (k+1)th candidate image contains features of the model image with the (k+1)th compression ratio. The image processing system (X1) includes a search processing unit (S1), an acquisition unit (51), and a sub-exploration unit (52). The search processing unit (S1) performs multiple image searches. The acquisition unit (51) acquires relevant information indicating the relationship between the Nth candidate image and the sub-candidate image. The sub-exploration unit (52) acquires a sub-candidate image based on the Nth candidate image obtained through the Nth image search and the relevant information acquired by the acquisition unit (51), and adds the sub-candidate image to the reserve candidates for the Mth image search.

[0153] The above structure enables high-speed processing for determining the extracted image.

[0154] Not limited to the above-described manner, various structures (including variations) of the image processing system (X1) involved in the implementation can be embodied by image processing methods and programs.

Claims

1. An image processing method that performs multiple image searches to explore images containing features of a model image from an object image, i.e., extracts the image. Let K be a variable representing natural numbers, and N and M be constants representing natural numbers, where M is greater than N. In the first image search, from the object image with the first compression ratio, a first candidate image containing features of the model image with the first compression ratio is explored, and the first candidate image is added to the candidates of the extracted image. In the (k+1)th image search, from the object images with a compression ratio of (k+1) that is smaller than the kth compression ratio, a preliminary candidate image for the (k+1)th image search is selected, including a kth candidate image that serves as a candidate for the extracted image obtained through the kth image search. From the preliminary candidates for the (k+1)th image search, a (k+1)th candidate image containing features of the model image with the (k+1)th compression ratio is explored, and the (k+1)th candidate image is added to the new candidates for the extracted image. The image processing method includes: The search process involves performing the image search multiple times. The process involves obtaining relevant information representing the correlation between the Nth candidate image and its sub-candidate images; and... The sub-exploration process involves obtaining the Nth candidate image based on the Nth candidate image obtained through the Nth image search and the relevant information obtained through the acquisition process, and then adding the sub-candidate image to the preliminary candidates for the Mth image search. The relevant information includes information related to the rotation angle difference between the Nth candidate image and the sub-candidate image. The image processing method further includes: a generation process, which generates information related to the rotation angle difference based on the rotational symmetry of the subject in the model image.

2. The image processing method according to claim 1, wherein, The image processing method further includes: relevant information determination processing, wherein the relevant information is determined based on the user's operation.

3. The image processing method according to claim 1 or 2, wherein, The image processing method further includes: The range expansion process involves adding multiple rotated images obtained by rotating the k-th candidate image or the sub-candidate image in stages within a range from 0 degrees to below a set angle to the preliminary candidates used in the k+1-th image search. and The settings process acquires information related to the set angle.

4. The image processing method according to claim 3, wherein, The image processing method further includes: setting an angle to determine the processing, and determining the set angle based on the user's operation.

5. The image processing method according to claim 1 or 2, wherein, N＝1。 6. The image processing method according to claim 1 or 2, wherein, In the search process, the image search is performed more than three times.

7. The image processing method according to claim 1 or 2, wherein, The image processing method explores the extracted image based on a model image.

8. A non-transitory recording medium having a program recorded for causing one or more processors to execute the image processing method according to any one of claims 1 to 7.

9. An image processing system that performs multiple image searches to explore images containing features of a model image from object images, i.e., extracts the image. Let K be a variable representing natural numbers, and N and M be constants representing natural numbers, where M is greater than N. In the first image search, from the object image with the first compression ratio, a first candidate image containing features of the model image with the first compression ratio is explored, and the first candidate image is added to the candidates of the extracted image. In the (k+1)th image search, from the object images with a compression ratio of (k+1) that is smaller than the kth compression ratio, preliminary candidates for the (k+1)th image search are selected, including the candidates for the extracted images obtained through the kth image search. From the preliminary candidates for the (k+1)th image search, a (k+1)th candidate image containing features of the model image with the (k+1)th compression ratio is explored, and the (k+1)th candidate image is added to the new candidates for the extracted images. The image processing system includes: The search processing unit performs the image search multiple times; The acquisition unit acquires relevant information relating the Nth candidate image to the sub-candidate images; and The sub-exploration unit determines the sub-candidate image based on the Nth candidate image obtained through the Nth image search and the relevant information obtained by the acquisition unit, and adds the sub-candidate image to the preliminary candidates for the Mth image search. The relevant information includes information related to the rotation angle difference between the Nth candidate image and the sub-candidate image. The image processing system further includes a generation processing unit that generates information related to the rotation angle difference based on the rotational symmetry of the subject in the model image.

Images