Feature point registration device, feature point registration method, and image processing system

Abstract

To provide a feature point registration device capable of supporting the selection of a feature point to be registered as a template by visualizing the variation of feature points extracted from each of the multiple images of an object.SOLUTION: The feature point registration device includes: a feature point extraction unit that extracts multiple feature points related to the object from each of multiple different input images in which the object is reflected; a feature comparison unit that compares the feature points extracted from each of multiple different input images among input images; and a registration unit that registers at least some of the multiple feature points extracted from at least one input image while associating with the object based on the comparison result of the feature points.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a feature point registration device, a feature point registration method, and an image processing system.

Background Art

[0002] In a production process in a factory, when it is necessary to pick a predetermined industrial part among a plurality of objects (for example, industrial parts used in the production of industrial products) flowing on a belt conveyor or the like, for example, it is required to quickly determine whether the object shown in the image captured by a camera is the object to be picked by processing the image. Conventionally, as such a determination method, a feature amount of a feature point of an object shown in an image captured by a camera is extracted, and it is determined whether it is an object to be picked by comparing (matching) with a feature amount of a predetermined feature point (so-called template).

[0003] Patent Document 1 discloses a technique for extracting a feature vector of each of a template and a search target image by applying a blurring process to each of the template and the search target image, performing a differentiation process on the blurred image, and integrating and histogramming the obtained edge intensity feature and edge direction feature within a predetermined region.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to Patent Document 1, it is possible to extract features (i.e., characteristic parts corresponding to feature points) from both the template and the image to be searched. However, it is not intended to determine whether the extracted feature points are appropriate feature points (for example, feature points that appear invariably in other similar individuals). In the aforementioned factory production process, objects (e.g., industrial parts) that enter the camera's field of view are similar because they have the same or similar model numbers, but such industrial parts can have variations in individual differences. Therefore, the feature points obtained by image processing of objects that enter the field of view one after another may vary from individual to individual and are not all appropriate, making it difficult to distinguish between extracted feature points that should be registered and those that should not, thus requiring higher efficiency in feature point registration.

[0006] This disclosure aims to provide a feature point registration device, a feature point registration method, and an image processing system that, in light of conventional circumstances, visualize the variation in feature points extracted from each of multiple images of an object and assist in selecting feature points to be registered as a template. [Means for solving the problem]

[0007] This disclosure includes a feature point extraction unit that extracts a plurality of feature points relating to an object from each of a plurality of different input images in which the object is depicted, and a feature comparison unit that compares the feature points extracted from each of the plurality of different input images between the input images. An operation reception unit that receives user input regarding feature points, Based on the comparison results of the aforementioned feature points, If the above operation is accepted, The present invention provides a feature point registration device comprising: a registration unit that registers some of the feature points extracted from at least one of the input images in association with the object.

[0008] Furthermore, this disclosure relates to a feature point registration method performed by a feature point registration device, comprising the steps of: inputting a plurality of different input images in which an object is shown; extracting a plurality of feature points relating to the object from each of the plurality of different input images; and comparing the feature points extracted from each of the plurality of different input images among the input images. A step to accept user input regarding feature points, Based on the comparison results of the aforementioned feature points, If the above operation is accepted, The present invention provides a feature point registration method comprising the step of registering some of the feature points extracted from at least one input image in association with the object.

[0009] Furthermore, this disclosure includes a camera capable of imaging an object, and a feature point registration device that is communicably connected to the camera, wherein the feature point registration device includes a feature point extraction unit that extracts a plurality of feature points relating to the object from each of a plurality of different input images in which the object is captured, and a feature comparison unit that compares the feature points extracted from each of the plurality of different input images between the input images. An operation reception unit that receives user input regarding feature points, Based on the comparison results of the aforementioned feature points, If the above operation is accepted, The present invention provides an image processing system comprising: a registration unit that registers some of the feature points extracted from at least one of the input images in association with the object. [Effects of the Invention]

[0010] According to this disclosure, it is possible to visualize the variation in feature points extracted from each of multiple images of the subject, and to support the selection of feature points that should be registered as a template. [Brief explanation of the drawing]

[0011] [Figure 1] A simplified diagram showing an example of the system configuration of an image processing system. [Figure 2] Block diagram showing a detailed internal configuration example of the image processing device according to Embodiments 1 and 2. [Figure 3] A flowchart illustrating an example of the operation procedure for calculating the feature deviation of feature points by an image processing device. [Figure 4] Flowchart showing an example of the operation procedure for feature point registration using the image processing device according to Embodiment 1. [Figure 5] This figure shows an example of a feature point distribution screen with a feature point image before adjustment operations on the adjustment bar. [Figure 6] This figure shows an example of a feature point distribution screen with an updated feature point image after adjustment operations on the adjustment bar. [Figure 7] A diagram showing a detailed example of the parameter adjustment area for image filters. [Figure 8] Flowchart showing an example of the operation procedure for feature point registration using the image processing device according to Embodiment 2. [Figure 9] Figure 7 shows an example of a feature point distribution screen displayed on the display device during operation. [Modes for carrying out the invention]

[0012] (Background leading to Embodiment 1) According to Japanese Patent Publication No. 2005-339075, it is possible to extract features (i.e., characteristic parts corresponding to feature points) from both the template and the image to be searched. However, it is not intended to determine whether the extracted feature points are appropriate feature points (for example, feature points that appear invariably in other similar individuals). In the aforementioned factory production process, objects (e.g., industrial parts) that enter the camera's field of view are similar because they have the same or similar model numbers, but such industrial parts can have variations in individual differences. Therefore, the feature points obtained by image processing of objects that successively enter the field of view may vary from individual to individual and are not all appropriate, making it difficult to distinguish between extracted feature points that should be registered and those that should not, thus requiring higher efficiency in feature point registration.

[0013] The following Embodiment 1 describes an example of a feature point registration device, a feature point registration method, and an image processing system that visualize the variation of feature points extracted from each of multiple images of an object and assist in selecting feature points to be registered as a template.

[0014] On the other hand, according to Japanese Unexamined Patent Application Publication No. 2005-339075, it is possible to extract the features of each of the template and the search target image (i.e., the characteristic parts corresponding to the feature points). However, it is not assumed to determine whether the extracted feature points are appropriate feature points (for example, feature points that invariably appear in another similar individual). In the production process in the factory described above, although objects (for example, industrial parts) that enter the imaging angle of the camera have the same or similar model numbers and similar ones flow in, there may be variations in individual differences in such industrial parts. Therefore, the feature points obtained by image processing of the objects that continuously enter the imaging angle may vary depending on the individual and are not necessarily all appropriate. Thus, it is difficult to distinguish between the feature points that can be registered and those that cannot, and higher efficiency is required in the registration of feature points. In particular, it is considered that more efficient feature point registration can be achieved if the registration of feature points can be confirmed from the perspective of the user (operator).

[0015] In the following Embodiment 1, an example of a feature point registration device, a feature point registration method, and an image processing system that visualize the feature points extracted from each of a plurality of images in which an object is photographed so as to be viewable by the user and assist the user in selecting the feature points to be registered as a template will be described.

[0016] Hereinafter, embodiments specifically disclosing a feature point registration device, a feature point registration method, and an image processing system according to the present disclosure will be described in detail with appropriate reference to the accompanying drawings. However, a more detailed description may be omitted as necessary. For example, a detailed description of well-known matters and a redundant description of substantially the same configuration may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims by these.

[0017] (Embodiment 1) In Embodiment 1, multiple images of an object (e.g., part BH) captured by camera 1, etc., are input to image processing device 10. The image processing device 10 compares feature points extracted from each image or from a transformed image obtained by applying predetermined image processing to that image. The image processing device 10 displays the comparison results in a manner that allows identification of feature points that have a high probability of appearing invariably in any image or transformed image and feature points that have appeared uniquely, using a method such as color coding. Based on the operation of user WK1, it determines whether or not to register the feature points to be used for actual pattern matching. Furthermore, if necessary, the image processing device 10 modifies (adjusts) the parameter values ​​of the image processing performed by image filter 13 or changes the parameter items based on the operation of user WK1, and displays the adjusted or changed parameters in a manner that allows identification of feature points that have a high probability of appearing invariably and feature points that have appeared uniquely.

[0018] In the following explanation, parameter adjustment is defined as modifying the value of a parameter used in the image processing performed by the image filter 13 (for example, the value of σ, which indicates the strength of the Gaussian filter). On the other hand, parameter change is defined as changing the type of parameter used in the image processing performed by the image filter 13 (for example, σ, which indicates the strength of the Gaussian filter used for blurring or sharpening) to another type of parameter (for example, the cutoff frequency, which indicates the pass frequency in the frequency domain of the pixel components).

[0019] Figure 1 is a simplified diagram showing an example of the system configuration of the image processing system 100. Embodiment 1 describes an example use case in which the image processing system 100 is used in a production process within a factory. The image processing system 100 includes a camera 1, an image processing device 10, an operating device 20, a display device 30, and a robot 40. The camera 1 and the image processing device 10, the image processing device 10 and the operating device 20, and the image processing device 10 and the display device 30 are connected in such a way that data signals can be input and output (sent and received).

[0020] Camera 1 is configured to have a field of view (field of view AG1) that captures objects (for example, industrial products or industrial parts, such as component BH; the same applies hereinafter) moving along the conveyor belt BLC1 installed in the factory. Camera 1 captures images of the objects (components BH) within the field of view AG1 at a predetermined frame rate, and sends the image data (i.e., input image) of each component BH obtained each time an image is captured to the image processing device 10.

[0021] The image processing device 10 (an example of a feature point registration device) is composed of a computer capable of performing predetermined processing (see below) using an input image of an object (part BH) captured by the camera 1. For example, it may be a PC (Personal Computer) or dedicated hardware equipment specialized for image processing. The image processing device 10 receives the input image of the object (part BH) captured by the camera 1, performs predetermined processing (see below) using the input image, generates a screen (see, for example, Figures 5, 6, or 9) as the processing result, and outputs (displays) it to the display device 30. The image processing device 10 may also perform predetermined processing (see below) based on a signal from the operating device 20 that detects the operation of user WK1. If the image processing device 10 extracts position information of part BH from the input image, it may generate a control signal including that position information and send it to the robot 40.

[0022] The operating device 20 is an interface that detects user WK1's input and is composed of, for example, a mouse, keyboard, or touch panel. When the operating device 20 receives an operation from user WK1, it generates a signal based on that operation and sends it to the image processing device 10.

[0023] The display device 30 is a device that outputs (displays) a screen for display (see, for example, Figures 5, 6, or 9) generated by the image processing device 10, and is composed of, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) device.

[0024] The robot 40 is equipped with a manipulator having a multi-jointed arm in which each of several links is rotatably connected via joints, and drives the tip of the manipulator (e.g., an end effector HN1 such as a robot hand) based on a control signal from the image processing device 10 (see above) to pick up a component BH or to perform alignment such as fine-tuning the mounting position of the component BH on a circuit board on which it is already mounted. Here, the robot 40 is exemplified as picking up and aligning a component BH based on a control signal from the image processing device 10, but it goes without saying that the actions of the robot 40 are not limited to these.

[0025] Figure 2 is a block diagram showing a detailed internal configuration example of an image processing device 10 according to Embodiments 1 and 2. The image processing device 10 includes a communication interface 11, a memory 12, an image filter 13, a feature point location extraction unit 14, a feature quantity calculation unit 15, a feature quantity selection unit 16, a feature comparison unit 17, a drawing unit 18, an image memory M1, a feature quantity memory M2, and a feature point memory M3. The communication interface 11, the memory 12, the image filter 13, the feature point location extraction unit 14, the feature quantity calculation unit 15, the feature quantity selection unit 16, the feature comparison unit 17, the drawing unit 18, the image memory M1, the feature quantity memory M2, and the feature point memory M3 are connected to each other via a data transmission bus (not shown) so that data signals can be input and output from each other.

[0026] The communication interface 11 (an example of an input interface) is a communication circuit that performs input and output (transmission and reception) of data signals between the camera 1 and the image processing device 10, between the image processing device 10 and the operation device 20, and between the image processing device 10 and the display device 30. The communication interface 11 receives imaging data (for example, an image showing component BH) input from the camera 1 each time and stores it in the image memory M1. The communication interface 11 also receives a signal from the operation device 20 corresponding to an adjustment or change operation of parameters used by the image filter 13 and sends that signal to the image filter 13. The communication interface 11 also receives a signal from the operation device 20 corresponding to a selection operation of feature points or their feature quantities extracted based on the input image showing component BH and sends that signal to the feature quantity selection unit 16. The communication interface 11 also sends various screens (for example, see Figures 5, 6, or 9) generated by the drawing unit 18 to the display device 30.

[0027] Memory 12 includes, for example, RAM (Random Access Memory) and ROM (Read Only Memory), and temporarily stores programs and control data necessary for the operation of the image processing device 10, as well as data generated or acquired by each part of the image processing device 10 during processing. RAM is, for example, work memory used by each part of the image processing device 10 during processing. ROM pre-stores, for example, programs and control data that define the processing of each part of the image processing device 10.

[0028] Image memory M1 is, for example, flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive), and stores multiple image data (for example, input images showing component BH) that are continuously input from camera 1.

[0029] The image filter 13 applies predetermined image processing to each of the multiple input images read from the image memory M1 using set values, which are parameters for image processing (for example, initial parameter values, or parameters adjusted or modified by the operating device 20). The image filter 13 is a Finite Impulse Response (FIR) filter, for example, composed of multi-stage taps, in which the filter coefficients of each tap can be freely set. The updating of the filter coefficients is achieved by a predetermined algorithm (for example, LMS (Least Mean Square)). The input image processed by the image filter 13 (hereinafter referred to as the "converted image") is input to the feature point position extraction unit 14. Here, the predetermined image processing may be, for example, a process that applies blurring (so-called smoothing or smoothing processing to smooth out changes in pixel values) to the input image, a process that applies sharpening (for example, a process that increases the contrast of contour parts in the input image), a process that emphasizes the gain of edge parts in the input image, or a process that reduces pixel components other than those in a predetermined frequency band using a bandpass filter. Furthermore, if the image filter 13 obtains a signal corresponding to an adjustment operation of the kernel size indicating the image processing unit by the operating device 20, it may also perform image processing using the kernel size after the adjustment operation. It has been explained that the image filter 13 applies predetermined image processing to each of the multiple input images read from the image memory M1, but it is also acceptable for each of the multiple input images read from the image memory M1 to be input directly to the feature point position extraction unit 14 without this image processing being applied. In this case, in the following explanation, the input image input to the feature point position extraction unit 14, even if the predetermined image processing is not applied by the image filter 13, may be referred to as a converted image.

[0030] The feature point location extraction unit 14 (an example of a feature point extraction unit) is composed of, for example, a DSP (Digital Signal Processor) or an FPGA (Field Programmable Gate Array). The feature point location extraction unit 14 targets the transformed image input from the image filter 13 and extracts multiple feature points related to the object (e.g., part BH) from the transformed image (i.e., the positions of feature points that indicate the maximum rotational change, scale change, or brightness change in the transformed image showing part BH). For this extraction method, known algorithms such as SIFT (Scale-Invariant Feature Transform) can be used. The feature point location extraction unit 14 extracts multiple feature points from a single input transformed image and sends the extraction result (e.g., point cloud data indicating the position of each of the multiple feature points in the transformed image) to the feature calculation unit 15. The point cloud data indicating the positions of feature points in a single processed image may be temporarily stored, for example, in memory 12 or in feature memory M2. The feature point position extraction unit 14 may also directly send the extraction results of multiple feature points related to the object (e.g., part BH) from the processed image to the feature comparison unit 17. In this case, the feature extraction results are compared between the processed images by the feature comparison unit 17 without the calculation of feature quantities, and the same applies hereafter.

[0031] The feature calculation unit 15 is configured, for example, by a DSP or FPGA. The feature calculation unit 15 uses the point cloud data of feature points for each transformed image input from the feature point location extraction unit 14 to calculate a feature (feature vector) consisting of a data sequence of numerical values ​​corresponding to each feature point. Once the feature calculation unit 15 has calculated the feature corresponding to the feature point for each transformed image, it stores the calculation result (data) of the feature corresponding to the feature point for each transformed image in the feature memory M2.

[0032] The feature memory M2 is, for example, flash memory, an HDD, or an SSD, and stores the calculation results (data) of feature quantities corresponding to the feature points for each transformed image calculated by the feature calculation unit 15.

[0033] The feature selection unit 16 (an example of a registration unit) is composed of, for example, a CPU (Central Proceeding Unit). The feature selection unit 16 reads the calculation results of feature points or corresponding feature quantities obtained for each of the multiple transformed images stored in the feature memory M2, and selects at least one of the feature points or feature quantities obtained for each of the transformed images based on the signal from the communication interface 11 and sends it to the drawing unit 18. The feature selection unit 16 registers (saves) feature point data indicating data such as the position of the selected feature point in one of the transformed images in the feature memory M3.

[0034] The feature comparison unit 17 is configured, for example, by a DSP or FPGA. The feature comparison unit 17 compares the calculated feature quantities (data) of each feature point in multiple transformed images, which are stored in the feature memory M2, and calculates the variation in feature quantities for each feature point (in other words, the feature deviation). Details of the feature deviation calculation process by the feature comparison unit 17 will be described later with reference to Figure 3. The feature comparison unit 17 temporarily stores the calculated feature deviation of feature points between transformed images in memory 12 or feature memory M2.

[0035] The drawing unit 18 (an example of an output unit) is composed of, for example, a CPU. The drawing unit 18 uses a single transformed image selected by the feature selection unit 16, feature point data indicating the positions of feature points in that transformed image, and the result of the feature deviation calculation of feature points between comparison images by the feature comparison unit 17 to generate a feature point image (see, for example, the feature point image SEL1 shown in Figure 5) by superimposing the feature points extracted from the transformed image onto the transformed image, and generates a screen (see, for example, Figures 5, 6, or 9) that includes this feature point image. The drawing unit 18 outputs (displays) the generated screen to the display device 30 via the communication interface 11.

[0036] The feature point memory M3 is, for example, flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive), and stores feature point data indicating the positions of feature points in a single converted image selected by the feature quantity selection unit 16 for each converted image.

[0037] Figure 3 is a flowchart illustrating an example of the operation procedure for calculating the feature deviation of feature points by the image processing device 10. In Figure 3, the image processing device 10 uses multiple images (for example, N (an integer of 2 or more)) input from the camera 1 to quantitatively calculate the extent of feature deviation between the images, based on the feature amounts of the feature points extracted from each image.

[0038] In Figure 3, the image filter 13 sets the variable i (a variable from 0 to (N-1)) to 0 (step St1) and reads the i-th input image from the image memory M1 (step St2). The image filter 13 applies a predetermined image processing (image filtering) to the i-th input image read in step St2 using the parameter x, which is the current setting value, and generates the i-th transformed image, which is the input image to which the image processing has been applied (step St3). As described above, image filtering may include, for example, blurring or sharpening.

[0039] The feature point location extraction unit 14 targets the i-th transformed image generated by the image filter 13 in step St3 and extracts multiple feature points related to the object (e.g., part BH) from the transformed image for each transformed image (step St4). The feature quantity calculation unit 15 uses the point cloud data of the feature points of the i-th transformed image extracted by the feature point location extraction unit 14 in step St4 to calculate a feature quantity C(i,k) (feature vector) consisting of a data sequence of numerical values ​​corresponding to each feature point (step St5). The feature quantity calculation unit 15 stores the feature quantity C(i,k) calculated in step St5 in the feature quantity memory M2 (step St6) and increments the current variable i (i←i+1) (step St7).

[0040] Here, we denote the feature quantity corresponding to the feature points of the i-th transformed image as C(i,k). k is the number of feature points extracted in the i-th transformed image, and is an integer greater than or equal to 2, but is a value that varies depending on the transformed image, and so on. For example, even if the value of k, which is the number of feature points extracted in the first transformed image showing part BH, is 10, the value of k, which is the number of feature points extracted in the second transformed image showing the same part BH, could be 10, 9, 11, or some other number. Therefore, k is not necessarily a fixed value, but a value that varies depending on the transformed image. Thus, C(i,k) represents a feature quantity (feature vector) composed of a data sequence of numbers that represent the feature quantities corresponding to each of the k feature points extracted in the i-th transformed image.

[0041] After step St7, the feature comparison unit 17 determines whether the variable i, incremented in step St7, has exceeded (N-1) (step St8). In other words, the process from step St2 to step St8 is repeated until the variable i exceeds (N-1) (step St8, NO). On the other hand, if the feature comparison unit 17 determines that the variable i has exceeded (N-1) (step St8, YES), it reads and obtains the feature quantities C(0,k), ..., C((N-1),k) calculated by repeating the process from step St2 to step St8 N times from the feature quantity memory M2 (step St9).

[0042] The feature comparison unit 17 compares the feature quantities C(0,k), ..., C((N-1),k) obtained in step St9 between the transformed images. In other words, the feature comparison unit 17 calculates the variation (in other words, the feature deviation) between the feature quantity C(0,k) corresponding to the feature point extracted in the first (i=0) transformed image, the feature quantity C(1,k) corresponding to the feature point extracted in the second (i=1) transformed image, ..., and the feature quantity C((N-1),k) corresponding to the feature point extracted in the Nth (i=(N-1)) transformed image (step St10).

[0043] In step St10, the feature comparison unit 17 calculates the vector distance (e.g., Euclidean distance) between feature quantities C calculated for each transformed image as an example of feature deviation. Note that the example of feature deviation is not limited to the vector distance between feature quantities C; the feature comparison unit 17 may also use the deviation value (in other words, variance) of the vector distance between feature quantities C calculated for each transformed image.

[0044] Furthermore, the feature comparison unit 17 may use a predetermined threshold value stored in the memory 12 to select only the feature quantities C whose vector distance (e.g., Euclidean distance) calculated for each transformed image is within that predetermined threshold value, and calculate the vector distance (e.g., Euclidean distance) or its variance between only the selected feature quantities C as an example of the feature deviation. This allows the feature comparison unit 17 to remove the influence of transformed images in which noise has clearly been added to the feature quantities C calculated for each of the N transformed images, and to calculate the feature deviation, including whether or not there is variation in the feature quantities C, more appropriately.

[0045] Furthermore, the feature comparison unit 17 may use a predetermined threshold and a weight coefficient for each distance (not shown) pre-stored in memory 12 to multiply each vector distance (e.g., Euclidean distance) between feature quantities C by a corresponding weight coefficient (for example, a smaller weight coefficient for longer distances and a larger weight coefficient for shorter distances). It may then select only the feature quantities C whose multiplication result exceeds the predetermined threshold and calculate the vector distance (e.g., Euclidean distance) or its variance between only the selected feature quantities C as an example of feature deviation. In this way, the feature comparison unit 17 can remove the influence of the transformed image where noise has clearly been added to the feature quantities C calculated for each of the N transformed images, and further consider the weight coefficient for each distance to make it easier to preferentially select feature quantities C with shorter distances (in other words, high similarity between feature points), thereby more appropriately calculating the variability of feature quantities C.

[0046] Figure 4 is a flowchart showing an example of the operation procedure for feature point registration by the image processing device 10 according to Embodiment 1. Figure 5 is a diagram showing an example of a feature point distribution screen having the feature point image SEL1 before adjustment operation on the adjustment bar BR1. Figure 6 is a diagram showing an example of a feature point distribution screen having the updated feature point image SEL2 after adjustment operation on the adjustment bar BR1. In the explanation of Figure 4, refer to Figures 5 and 6 as needed.

[0047] For example, as shown in Figures 5 and 6, we illustrate the case where N, which indicates the number of images to be processed by the comparison process (see below) in the image processing device 10, is 5. In other words, in Embodiment 1, it is assumed that five input images, CTG1, CTG2, CTG3, CTG4, and CTG5, are input from the camera 1 to the image processing device 10.

[0048] In Figure 4, the image processing device 10 (e.g., image filter 13) obtains the parameter x currently set in the image filter 13 from the image filter 13 (step St11). The image processing device 10 (e.g., image filter 13) generates a transformed image corresponding to each of the multiple input images (e.g., N=5) input from the camera 1. The image processing device 10 (e.g., feature comparison unit 17) compares the feature quantities C(0,k), ..., C(4,k) corresponding to the feature points extracted in each transformed image between the transformed images.

[0049] In other words, the image processing device 10 (for example, the feature comparison unit 17) calculates the variation (in other words, the feature deviation) between the feature quantity C(0,k) corresponding to the feature point extracted in the first (i=0) transformed image, the feature quantity C(1,k) corresponding to the feature point extracted in the second (i=1) transformed image, ... and the feature quantity C(4,k) corresponding to the feature point extracted in the fifth (i=4) transformed image (step St12). The process in step St12 is the same as the process shown in Figure 3, so a detailed explanation is omitted. In other words, the process in step St12 in Figure 4 is a subroutine, and the details of the process of that subroutine are shown in Figure 3.

[0050] The image processing device 10 (e.g., the drawing unit 18) selects a specific one of the five transformed images based on the comparison of feature quantities C between the transformed images in step St12. Furthermore, the image processing device 10 (e.g., the drawing unit 18) generates a feature point image SEL1 (see, for example, Figure 5) by superimposing the feature points extracted in the specific transformed image onto that transformed image, based on the feature quantity deviation between the feature quantities corresponding to the feature points extracted in the selected specific transformed image and the feature quantities corresponding to the feature points extracted in each of the other four transformed images. It then generates a feature point distribution screen WD1 (see, Figure 5) including this feature point image SEL1 and outputs (displays) it to the display device 30 (step St13).

[0051] For example, the feature point image SEL1 shown in Figure 5 is displayed along with an indicator P1 that shows the parameter x obtained in step St11 (for example, the sigma (σ) of the Gaussian filter, which is a parameter set when blurring, is 1.0). The feature point image SEL1 displays extracted feature points in two color-coded stages based on the feature deviation with the corresponding feature quantities extracted in other transformed images. For example, feature points whose feature deviation is smaller than a predetermined value stored in memory 12 and are likely to appear invariantly in other transformed images are shown in orange (represented as a white-filled rectangle in Figure 5). These orange (white-filled rectangle) feature points have the attribute of being feature points that should be registered (saved). On the other hand, feature points that are unique to that transformed image and are unlikely to appear in other transformed images, and whose feature deviation is larger than a predetermined value stored in memory 12, are shown in blue (represented as a white-filled circle in Figure 5). These blue (white-filled circle) feature points have the attribute of being feature points that should not be registered (saved).

[0052] Furthermore, the feature point distribution screen WD1 in Figure 5 displays a parameter adjustment area PR1, which includes adjustment bars BR1 and BR2 that allow the user WK1 to adjust the type of parameter x set in the image filter 13.

[0053] The adjustment bar BR1 allows the user WK1 to specify whether to perform blurring (in other words, smoothing) or sharpening by sliding a knob TM1 horizontally. In other words, by sliding the knob TM1 (see Figure 6), it is possible to adjust the σ value of the Gaussian filter to a value suitable for blurring or to a value suitable for sharpening.

[0054] The adjustment bar BR2 allows the user WK1 to arbitrarily specify the magnitude of the threshold (for example, the threshold for feature deviation) using a knob TM2 that can be slid horizontally. In other words, by sliding the knob TM2, the user WK1 can arbitrarily adjust the threshold to select images that will be used for feature comparison processing, excluding transformed images that clearly contain noise.

[0055] The image processing device 10 (for example, the feature selection unit 16) may perform attribute editing on at least one feature point (for example, a unique feature point) among the multiple feature points superimposed on the feature point image SEL1 of the feature point distribution screen WD1 displayed in step St13, by operation using the user WK1's operating device 20 such as a mouse (step St14). Step St14 is optional and may be omitted. Note that attribute editing means, for example, changing the setting of a feature point that was displayed in a manner that makes it possible to identify it as a unique feature point, to a feature point that should be registered (saved) in step St17 (i.e., a feature point that is highly likely to appear invariably) by operation of the user WK1.

[0056] When the image processing device (e.g., feature selection unit 16) detects that the OK button Bt1 on the feature point distribution screen WD1 has been pressed as an operation by user WK1 (registration completion operation) (step St15, YES), it registers (saves) at least one feature point that has the attribute of being a feature point to be registered (saved) in step St13 or step St14, associating it with the identification information of the object (e.g., part BH) in the feature point memory M3 (step St17). This registered (saved) feature point is used in actual operation for matching (e.g., pattern matching) with feature points extracted from the input image input from camera 1 (step St18).

[0057] On the other hand, if the image processing device 10 detects that the NG button Bt2 on the feature point distribution screen WD1 has been pressed as an operation by user WK1 (parameter reset operation) (step St15, NO), the image filter 13 detects an operation by user WK1 (i.e., an adjustment or change operation of parameter x acquired in step St11) (step St16). After step St16, the image processing device 10 returns to step St11. In other words, the image processing device 10 repeats the processes from step St11 to step St14 until it detects that user WK1 has completed registration.

[0058] For example, in Figure 5, suppose the image filter 13 detects that user WK1 has slid the knob TM1 so that it tilts "smoothly" as an adjustment operation for parameter x (step St16). In this case, the image processing device 10 (for example, the image filter 13) obtains the parameter x after the adjustment operation (for example, from "σ=1.0" before adjustment to "σ=2.0" after adjustment, see Figure 6) (step St11). The image processing device 10 then uses the parameter x after the slide (i.e., after user WK1's adjustment operation for parameter x) to similarly execute the processes from step St12 to step St13, generating the feature point distribution screen WD1A shown in Figure 6 and outputting (displaying) it on the display device 30.

[0059] For example, the feature point image SEL2 shown in Figure 6 is displayed along with indicator P2, which indicates the parameter x acquired in step St11 (for example, the sigma (σ) of the Gaussian filter, which is a parameter set when blurring, is 2.0) after the parameter x was reset in step St16 (for example, by adjusting the value or changing the parameter itself). The feature point image SEL2 displays extracted feature points in two color-coded stages based on the feature deviation with the corresponding feature quantities extracted in other transformed images. For example, feature points whose feature deviation is smaller than a predetermined value stored in memory 12 and which are likely to appear invariantly in any transformed image are shown in orange (represented as a white-filled rectangle in Figure 6). These orange (white-filled rectangle) feature points have the attribute that they should be registered (saved). On the other hand, feature points that are unique to that transformed image and are unlikely to appear in other transformed images, and whose feature deviation is larger than a predetermined value stored in memory 12, are shown in blue (represented as a white-filled circle in Figure 6). These blue (white-filled circle) feature points have the attribute that they are feature points that will not be registered (saved).

[0060] In the feature point image SEL1 shown in Figure 5 and the feature point image SEL2 shown in Figure 6, the positions of the orange (white-filled square) feature points and the blue (white-filled square) feature points are slightly different. This is because the parameter x used to generate the feature point image SEL1 (e.g., σ=1.0) and the parameter x used to generate the feature point image SEL2 (e.g., σ=2.0) are different, resulting in a different degree of blurring of the image used for feature point extraction.

[0061] Figure 7 shows a detailed example of the parameter adjustment area PR2 of the image filter 13. The parameter adjustment area PR2 displays not only the adjustment bars BR1 and BR2 shown in Figures 5 and 6, respectively, but also the other adjustment bars BR3, BR4, BR5, BR6, BR7, and BR8. Note that the explanation of adjustment bars BR1 and BR2 is the same as that explained in Figure 5, so the explanation is omitted here.

[0062] The adjustment bar BR3 allows the user WK1 to operate a knob TM3 that slides horizontally, enabling the user to specify the lower limit of the cutoff frequency to either the low-frequency or high-frequency side. In other words, by sliding the knob TM3 to the low-frequency or high-frequency side (see Figure 7), it is possible to adjust the lower frequency limit of the frequency components that can pass through the bandpass filter to the desired value.

[0063] The adjustment bar BR4 allows the user WK1 to operate a knob TM4 that slides horizontally, enabling the user to specify the upper limit of the cutoff frequency to either the low-frequency or high-frequency side. In other words, by sliding the knob TM4 to the low-frequency or high-frequency side (see Figure 7), it is possible to adjust the upper frequency limit of the frequency components that can pass through the bandpass filter to the desired value.

[0064] The adjustment bar BR5 allows the user WK1 to specify the steepness of the filter coefficient characteristics of the image filter 13 (for example, the degree of steepness of the main lobe of the window function) to be gentler or steeper by sliding the knob TM5 horizontally. In other words, by sliding the knob TM5 to the gentler or steeper side (see Figure 7), it is possible to adjust the steepness of the filter coefficient characteristics of the image filter 13 to the desired value.

[0065] The adjustment bar BR6 allows the user WK1 to slide a knob TM6 horizontally to specify a lower or higher detection threshold for edge components. In other words, by sliding the knob TM6 lower or higher (see Figure 7), it is possible to adjust the detection threshold for edge components that are easily extracted as feature points to a desired value.

[0066] The adjustment bar BR7 allows the user WK1 to slide a knob TM7 horizontally to specify a lower or higher intensity (σ) of the Gaussian filter used by the image filter 13 for smoothing and other processes. In other words, by sliding the knob TM7 lower or higher (see Figure 7), it is possible to adjust the intensity (σ) of the Gaussian filter used by the image filter 13 when performing filtering to a desired value.

[0067] The adjustment bar BR8 allows the user WK1 to narrow or widen the kernel size, which defines the size of the image to be filtered by the image filter 13, by sliding the knob TM8 horizontally. In other words, by sliding the knob TM8 narrower or wider (see Figure 7), the kernel size can be adjusted to the desired value.

[0068] As described above, the image processing system 100 according to Embodiment 1 includes a feature point position extraction unit 14 that extracts multiple feature points relating to an object from each of a plurality of different input images (for example, input images CTG1, CTG2, CTG3, CTG4, CTG5) that show the object (for example, part BH), a feature comparison unit 17 that compares the feature points extracted from each of the plurality of different input images among the input images, and a feature quantity selection unit 16 that, based on the comparison result of the feature points, registers some of the multiple feature points extracted from at least one input image in association with the object.

[0069] As a result, the image processing device 10 can compare feature points extracted from each of the multiple input images CTG1 to CTG5 in which the object (e.g., part BH) is captured by camera 1, etc., and can support the registration of feature points that can be used for pattern matching with feature points extracted from input images showing part BH that are input from camera 1, etc. during actual operation.

[0070] Furthermore, the image processing device 10 includes a feature calculation unit 15 that calculates the feature quantities of multiple feature points extracted from each of the input images (for example, input images CTG1, CTG2, CTG3, CTG4, CTG5). The feature comparison unit 17 uses the calculation results of the feature quantities corresponding to the feature points extracted in the input images to calculate and compare the similarity of feature points between the input images (for example, a feature deviation indicating variability). As a result, the image processing device 10 can quantitatively visualize the variability of feature points extracted from each of the multiple input images CTG1 to CTG5, and can assist in selecting feature points that should be registered as templates.

[0071] Furthermore, the image processing device 10 includes an image filter 13 that applies image transformation processing to the input image using predetermined parameters (for example, parameter x). The feature point position extraction unit 14 extracts multiple feature points based on the transformed image, which is the input image after image transformation processing by the image filter 13. As a result, the image processing device 10 can improve the accuracy of feature point extraction in each of the multiple transformed images by using the transformed image that has been processed by the image filter 13.

[0072] Furthermore, if the feature selection unit 16 determines that the calculation result of the similarity of feature points between input images is below a threshold, it registers some of the feature points used for comparison between input images. As a result, the image processing device 10 can register feature points with low variability as having a high probability of appearing invariably in any input image when the similarity (feature deviation, which is the so-called variation) between feature points extracted from each of the multiple input images is small enough to be below a threshold.

[0073] Furthermore, the image processing device 10 includes a feature calculation unit 15 that calculates the feature quantities of each of the multiple feature points extracted from the input image. The feature comparison unit 17 uses the calculation results of the feature quantities corresponding to the feature points extracted in the transformed image to calculate and compare the similarity of feature points between the transformed images (for example, a feature quantity deviation indicating variability). As a result, the image processing device 10 can visualize the variability of feature points extracted from the transformed images, each of the multiple input images CTG1 to CTG5 that have been image processed, and can assist in selecting feature points that should be registered as a template.

[0074] Furthermore, after the number of transformed images for which feature quantities corresponding to feature points have been calculated reaches a predetermined value (for example, 5), the feature comparison unit 17 calculates and compares the similarity of feature points (for example, a feature quantity deviation indicating variability) among the transformed images of that predetermined number. As a result, the image processing device 10 can calculate with high reliability the variability that serves as an indicator of whether or not the feature points extracted in the transformed images should be registered, based on the comparison among the transformed images of that predetermined number.

[0075] Furthermore, the image processing device 10 includes a drawing unit 18 that generates a feature point image SEL1 by superimposing multiple feature points extracted by the feature point position extraction unit 14 from one of several different input images, and outputs a feature point distribution screen WD1, which includes at least the feature point image SEL1, to the display device 30. As a result, the image processing device 10 can visually output a feature point image SEL1 in which the feature points extracted from one of several input images are superimposed in a way that shows their positions, thus facilitating verification by a user WK1 who views this feature point image SEL1 visually or otherwise.

[0076] Furthermore, the drawing unit 18 outputs the feature point distribution screen WD1 to the display device 30, including multiple different input images (for example, input images CTG1 to CTG5) and the feature point image SEL1 in a way that allows for comparison. This allows user WK1 to easily compare the input image before it is filtered by the image filter 13 input from the camera 1, etc., with the feature point image SEL1 superimposed on it so that the positions of the feature points extracted from that input image can be seen, and to easily judge the quality of the feature points to be registered using the feature point distribution screen WD1.

[0077] Furthermore, the image processing device 10 includes a feature calculation unit 15 that calculates the feature quantities of each of the multiple feature points extracted from the input image. The feature comparison unit 17 uses the calculation results of the feature quantities corresponding to the feature points extracted in the input image to calculate and compare the similarity of feature points between input images (for example, a feature quantity deviation that indicates variability). The drawing unit 18 draws the feature points on the feature point image SEL1 by color-coding the calculation results of the similarity of feature points. This allows user WK1 to easily determine, by the color of the feature points superimposed on the feature point image SEL1, whether a feature point has a high probability of appearing invariantly (commonly) in any input image (i.e., a feature point suitable for registration) or a feature point that appeared specifically in the input image (i.e., a feature point unsuitable for registration).

[0078] Furthermore, the image conversion process is either smoothing or sharpening. This allows the image processing device 10 to improve the accuracy of extracting feature points suitable for registration.

[0079] Furthermore, the image processing device 10 is equipped with an image memory M1 that stores multiple different input images received from a camera 1 capable of capturing images of an object (e.g., a component BH). This allows the image processing device 10 to secure a data buffer for each input image received from the camera 1, etc., thereby suppressing buffer underflow within the image processing device 10.

[0080] Furthermore, the image processing device 10 is equipped with a feature memory M2 that stores the calculation results of feature quantities corresponding to feature points extracted from the input image. The feature comparison unit 17 uses the calculation results of feature quantities stored in the feature memory M2 to calculate the similarity of feature points between input images. As a result, the image processing device 10 can read the feature quantity data corresponding to the feature points from the feature memory M2, thereby effectively reducing the load of the feature quantity comparison process (in other words, the feature quantity deviation calculation process) between multiple input images.

[0081] Furthermore, the image processing device 10 is equipped with a communication interface 11 that accepts adjustment operations for predetermined parameters. The image filter 13 performs image transformation processing on the input image using the predetermined parameters after the adjustment operation. This allows the image processing device 10 to easily adjust the parameters of the image processing performed by the image filter 13 through parameter adjustment operations by the user WK1.

[0082] Furthermore, in the image processing system 100 according to Embodiment 1, the image processing device 10 includes an image filter 13 that applies image transformation processing using parameters to each of a plurality of different input images (e.g., input images CTG1 to CTG5) in which an object (e.g., part BH) is shown; a feature point position extraction unit 14 that extracts a plurality of feature points relating to the object from each of the plurality of different transformed images, which are input images after image transformation processing by the image filter 13; a drawing unit 18 that generates a feature point image SEL1 by superimposing the plurality of feature points extracted by the feature point position extraction unit 14 on one of the plurality of different transformed images, and outputs a feature point distribution screen WD1 including at least the feature point image and a parameter adjustment bar BR1 to the display device 30; and a communication interface 11 that accepts operations from user WK1. When a parameter change operation on the adjustment bar BR1 by user WK1 (e.g., an operation to adjust the value of a parameter) is input to the communication interface 11, the drawing unit 18 updates the feature point image based on the changed parameter and outputs the screen to the display device 30.

[0083] As a result, the image processing device 10 can visualize the feature points extracted from each of the multiple images of the object (e.g., part BH) in a way that is viewable to the user WK1, thereby assisting the user WK1 in selecting feature points that should be registered as a template.

[0084] Furthermore, the image processing device 10 includes a feature comparison unit 17 that compares feature points extracted from each of several different transformed processing images between the transformed processing images, and a feature quantity selection unit 16 that, based on the comparison results of the feature points, registers some of the multiple feature points of the feature point image SEL1 in association with the target object. As a result, the image processing device 10 can compare feature points extracted from each of several input images CTG1 to CTG5 in which the target object (e.g., part BH) is captured by the camera 1, etc., and can support the registration of feature points that can be used for pattern matching with feature points extracted from input images showing part BH that are input from the camera 1, etc. during actual operation.

[0085] Furthermore, the image processing device 10 includes a feature calculation unit 15 that calculates the feature quantities of each of the multiple feature points extracted from the transformed image. The feature comparison unit 17 uses the calculation results of the feature quantities corresponding to the feature points extracted in the transformed image to calculate and compare the similarity of feature points between the transformed images (for example, a feature quantity deviation that indicates variability). As a result, the image processing device 10 can quantitatively visualize the variability of feature points extracted from each of the multiple input images CTG1 to CTG5 between the input images.

[0086] Furthermore, the drawing unit 18 outputs multiple feature points in the feature point image SEL1 in at least two different colors, according to the calculation result of the similarity of feature points between the converted images. This allows user WK1 to easily determine, based on the color of the feature points superimposed on the feature point image SEL1, whether a feature point has a high probability of appearing invariantly (commonly) in any input image (i.e., a feature point suitable for registration) or a feature point that appears specifically in the input image (i.e., a feature point unsuitable for registration).

[0087] Furthermore, when a user WK1 inputs a specification operation to register or delete a feature point from among multiple feature points in the feature point image SEL1 into the feature point selection unit 16, the user WK1 registers or deletes the feature point specified by that operation. This allows the user WK1 to visually confirm the candidate feature points that should be registered in the feature point image SEL1, and then easily choose whether to actually register or delete those feature points, thereby improving the convenience of registering feature points.

[0088] (Background leading to Embodiment 2) According to Japanese Patent Publication No. 2005-339075, blurring is performed to extract features (i.e., characteristic parts corresponding to feature points) from both the template and the target image. However, it is not assumed to be the extent (in other words, what parameters) of blurring should be applied to both the template and the target image in order to obtain appropriate feature points (for example, feature points that appear invariably in other similar individuals). In the factory production process described above, objects (e.g., industrial parts) that enter the camera's field of view are similar because they have the same or similar model numbers, but such industrial parts can have variations in individual differences. Therefore, it is difficult to apply optimal blurring or other image processing to each object that enters the field of view one after another. However, if it is possible to apply blurring or other image processing using highly probable and efficient parameters to extract appropriate feature points (see above) regardless of which object enters the field of view, it is expected that the feature point registration process will become more efficient.

[0089] The following Embodiment 2 describes examples of an image processing device, an image processing method, and an image processing system that select appropriate filter coefficients for an image filter to be applied to each of multiple images of an object, and that assist in selecting feature points to be registered as a template.

[0090] (Embodiment 2) In Embodiment 1, an example was described in which the adjustment of the parameter x of the image filter 13 is mainly performed manually by the user WK1. In Embodiment 2, an example is described in which the adjustment of the parameter x of the image filter 13 is performed automatically within the image processing device 10. Specifically, in Embodiment 2, the image processing device 10 automatically adjusts M types of parameter values ​​(M: constant integers of 2 or more) used by the image filter 13 for image processing, calculates the variation in feature quantities (feature quantity deviation) of feature points extracted from multiple input images or transformed images using each value, and generates (M+1) feature point images corresponding to each of the M types of parameters based on any of the input images and displays them comparatively (see Figure 9).

[0091] The configuration of the image processing system according to Embodiment 2 is the same as the configuration of the image processing system 100 according to Embodiment 1. In describing Embodiment 2, the same reference numerals are used for components identical to those in Embodiment 1 to simplify or omit their explanations, while different components are described.

[0092] Figure 8 is a flowchart showing an example of the operation procedure for feature point registration by the image processing device 10 according to Embodiment 2. Figure 9 is a diagram showing an example of the feature point distribution screen WD2 displayed on the display device 30 during the operation of Figure 7. In the explanation of Figure 8, refer to Figure 9 as needed. In the explanation of Figure 8, the same step numbers are assigned to processes that are the same as those in Figure 4, and the explanation is simplified or omitted, while different content is explained.

[0093] For example, as shown in Figure 9, we illustrate the case where N, which indicates the number of images to be processed for comparison in the image processing device 10, is 5. In other words, in Embodiment 2 as well, we assume that five input images, CTG1, CTG2, CTG3, CTG4, and CTG5, are input from the camera 1 to the image processing device 10.

[0094] In Figure 8, the image processing device 10 (e.g., image filter 13) sets the variable j (a variable from 0 to M) to 0 (step St21), and sets the image filter 13 to use the j-th setting value for the parameter x used in image processing (step St22). The image processing device 10 (e.g., image filter 13) generates a transformed image corresponding to each of the multiple input images (e.g., N=5) input from camera 1, using the j-th setting value (parameter x). The image processing device 10 (e.g., feature comparison unit 17) compares the feature quantities C(0,j,k), ..., C(4,j,k) corresponding to the feature points extracted in each transformed image between the transformed images. Here, the feature quantity C(i,j,k) represents the feature quantity corresponding to the feature point extracted from the i-th transformed image generated by the image filter 13 using the j-th setting value (parameter x).

[0095] In other words, the image processing device 10 (for example, the feature comparison unit 17) calculates the variation (in other words, the feature deviation) between the feature quantity C(0,j,k) corresponding to the feature point extracted in the first (i=0) transformed image, the feature quantity C(1,j,k) corresponding to the feature point extracted in the second (i=1) transformed image, ... and the feature quantity C(4,j,k) corresponding to the feature point extracted in the fifth (i=4) transformed image (step St12). Since the process in step St12 is the same as the process shown in Figure 3, a detailed explanation is omitted. In other words, the process in step St12 in Figure 8 is a subroutine, and the details of the process of that subroutine are shown in Figure 3.

[0096] The image processing device 10 (for example, the feature comparison unit 17) stores the feature deviation calculated in step St12 as the feature deviation result corresponding to the j-th setting value (parameter x) in the feature memory M2 (step St23), and increments the current variable j (j←j+1) (step St24). After step St24, the image processing device 10 (for example, the feature comparison unit 17) determines whether the variable j after being incremented in step St24 has exceeded M (step St25). In other words, the processing in steps St22, St12, St23, St24, and St25 is repeated until the variable j exceeds M (step St25, NO). On the other hand, if the image processing device 10 (for example, the feature comparison unit 17) determines that the variable j exceeds M (step st25, YES), it reads the feature deviation result calculated by repeating the process in steps St22, St12, St23, St24, and St25 M times from the feature memory M2 and obtains it, and selects the parameter x corresponding to the feature deviation result that minimizes the deviation value (step St26).

[0097] The image processing device 10 (e.g., the drawing unit 18) selects a specific image (e.g., input image CTG1) from among the five input images CTG1 to CTG5. Based on each of the eight images, including the selected input image CTG1 (i.e., the original image that has not been processed by the image filter 13) and the transformed image generated by adjusting up to M (e.g., M=7) parameters x including the parameter x selected in step St26, the image processing device 10 (e.g., the drawing unit 18) generates feature point images JT1, JT2, JT3, JT4, JT5, JT6, JT7, and JT8 in the same manner as in Embodiment 1. The image processing device 10 (e.g., the drawing unit 18) generates a feature point distribution screen WD2 (see Figure 9) including these feature point images JT1 to JT8 and outputs (displays) it to the display device 30 (step St27).

[0098] In particular, it is preferable that in step St27, the image processing device 10 (e.g., the drawing unit 18) displays a registration recommendation frame WK0 in a recognizable manner (e.g., a red frame) around the feature point image (e.g., feature point image JT7) that has the smallest feature point deviation among the feature point images JT1 to JT8 (i.e., where the feature points to be registered appear most ideally), to indicate that the best feature points appear. After step St27, the processing of the image processing device 10 proceeds to step St15 (see Figure 4).

[0099] For example, the feature point image JT1 shown in Figure 9 is an image in which multiple feature points extracted from an input image (e.g., input image CTG1) input to the image processing device 10 are superimposed on that input image, and, as in Embodiment 1, the feature points are displayed in different colors according to their attributes. In other words, feature points whose feature deviation is smaller than a predetermined value stored in memory 12 and which are likely to appear invariantly in other transformed images are shown in orange (represented as white squares in Figure 9). These orange (white square) feature points have the attribute of being feature points that should be registered (saved). On the other hand, feature points that are unique to that transformed image and are extracted when the feature deviation is larger than a predetermined value stored in memory 12 and which are unlikely to appear in other transformed images are shown in blue (represented as white circles in Figure 9). These blue (white circle) feature points have the attribute of being feature points that should not be registered (saved).

[0100] Furthermore, the feature point image JT2 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is obtained by image processing using the parameter x (i.e., σ=1.0) on an input image (for example, input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0101] Furthermore, the feature point image JT3 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is processed using the parameter x (i.e., σ=2.0) on an input image (e.g., input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0102] Furthermore, the feature point image JT4 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is obtained by image processing using the parameter x (i.e., σ=3.0) on an input image (e.g., input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0103] Furthermore, the feature point image JT5 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is obtained by image processing using the parameter x (i.e., σ=4.0) on an input image (for example, input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0104] Furthermore, the feature point image JT6 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is obtained by image processing using the parameter x (i.e., σ=5.0) on an input image (for example, input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0105] Furthermore, the feature point image JT7 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is obtained by image processing using the parameter x (i.e., σ=6.0) on an input image (for example, input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0106] Furthermore, the feature point image JT8 shown in Figure 9 is an image in which multiple feature points extracted from a transformed image, which is obtained by image processing using the parameter x (i.e., σ=7.0) on an input image (for example, input image CTG1) input to the image processing device 10, are superimposed on that transformed image. Similar to Embodiment 1, the feature points are displayed in different colors according to their attributes. The manner of color coding is the same as described above, so a detailed explanation is omitted.

[0107] The image processing device 10 (for example, the drawing unit 18) displays a parameter adjustment area PR2 on the feature point distribution screen WD2 (see Figure 9), which includes an adjustment bar BR2 for arbitrarily specifying the magnitude of the feature deviation threshold. The adjustment bar BR2 allows the user WK1 to arbitrarily specify the magnitude of the threshold (for example, the feature deviation threshold) using a knob TM2 that can be slid horizontally. In other words, by sliding the knob TM2, the user WK1 can arbitrarily adjust the threshold to select images to be used for feature comparison processing of feature points, while excluding converted images that clearly contain noise.

[0108] Returning to Figure 8, the image processing device 10 (for example, the feature selection unit 16) detects that the OK button Bt1 on the feature point distribution screen WD2 has been pressed as an operation by user WK1 (registration completion operation) (step St15, YES). In step St26, it registers the parameter x selected in step St26 into the image filter 13, and in step St27, it registers (saves) at least one feature point that has the attribute of being a feature point to be registered (saved) among the feature points superimposed on the feature point image in which the registration recommendation frame WK0 is displayed, in association with the identification information of the object (for example, part BH) into the feature point memory M3 (step St28). This registered (saved) feature point is used in actual operation for matching with feature points extracted in the input image input from camera 1 (for example, pattern matching) (step St18).

[0109] On the other hand, if the image filter 13 detects that the NG button Bt2 on the feature point distribution screen WD2 has been pressed as an operation by user WK1 (parameter reset operation) (step St15, NO), the image filter 13 detects the operation by user WK1 (i.e., the operation to change parameter x obtained in step St22) (step St29). After step St29, the image processing device 10 returns to step St21. In other words, the image processing device 10 repeats the processes up to steps St21, St22, St12, St23, St24, St25, St26, and St27 until it detects that user WK1 has completed registration.

[0110] As described above, the image processing system 100 according to Embodiment 2 includes an image filter 13 that applies image transformation processing to each of a plurality of different input images (e.g., input images CTG1 to CTG5) showing an object (e.g., part BH) using one different parameter from a plurality of parameters; a feature point position extraction unit 14 that extracts a plurality of feature points related to the object from each of the plurality of different transformed images, which are input images after image transformation processing by the image filter 13; a feature comparison unit 17 that compares the feature points extracted from each of the plurality of different transformed images among the transformed images; and a feature quantity selection unit 16 that registers a specific parameter, which is one of a plurality of parameters, as a filter coefficient for matching processing to the image filter 13 based on the comparison result of the feature points.

[0111] This allows the image processing device 10 to select appropriate filter coefficients for the image filter to be applied to each of the multiple input images in which the object (e.g., part BH) is photographed, and to assist in selecting feature points that should be registered as a template.

[0112] Furthermore, the feature selection unit 16 registers some of the feature points extracted from the transformed image generated by the image filter 13 using specific parameters, associating them with the target object (e.g., part BH). As a result, the image processing device 10 can register some of the feature points extracted from the transformed image after image processing using the optimal values ​​determined as parameters used by the image filter 13 for image processing, with high reliability.

[0113] Furthermore, the image processing device 10 includes a feature calculation unit 15 that calculates the feature quantities of each of the multiple feature points extracted from the transformed image. The feature comparison unit 17 uses the calculation results of the feature quantities corresponding to the feature points extracted in the transformed image to calculate and compare the similarity (e.g., feature deviation) of feature points between the transformed images. As a result, the image processing device 10 can quantitatively visualize the variation in feature points extracted from each of the multiple transformed images, and can assist in selecting feature points that should be registered as a template.

[0114] Furthermore, the feature selection unit 16 registers specific parameters in the image filter 13 if the calculation result of the similarity of feature points between transformed images generated by the image filter 13 using specific parameters is below a threshold. This allows the image processing device 10 to register useful parameters of the image filter 13 that contributed to the appearance of feature points with small similarity (feature deviation, which is the so-called variation) between feature points extracted in each of the multiple transformed images, so that they can be used in the image filter 13 when the similarity (feature deviation, which is the so-called variation) between feature points is small enough to be below a threshold.

[0115] Furthermore, if the feature comparison unit 17 determines that the similarity of feature points between transformed images generated by the image filter 13 using specific parameters is greater than a threshold, it changes the specific parameter to another parameter from among several parameters and sets it in the image filter 13. This allows the image processing device 10 to quickly select other useful parameters by treating the parameter used to extract the feature point as useless and prohibiting its use if the result obtained is that the feature deviation is greater than the threshold.

[0116] Furthermore, the feature selection unit 16 registers the parameter that minimizes the sum of the similarity calculation results of feature points between transformed processed images generated by the image filter 13 using one different parameter from among several parameters in the image filter 13. This allows the image processing device 10 to register the parameter of the image filter 13 that contributed to the calculation of the feature when the sum of the feature deviations calculated for multiple transformed processed images is minimized, making it available for use in the image filter 13 as a useful parameter.

[0117] Furthermore, the feature selection unit 16 registers parameters in the image filter 13 such that the sum of the similarity calculation results of feature points between transformed processed images, each generated by the image filter 13 using one different parameter from a set of multiple parameters, is less than or equal to a predetermined value. As a result, the image processing device 10 can register the parameters of the image filter 13 that contributed to the calculation of the feature as useful parameters for use in the image filter 13, even if the sum of the feature deviations calculated for multiple transformed processed images is not the minimum but is small enough to be less than or equal to a predetermined value.

[0118] Furthermore, the image processing device 10 generates multiple feature point images JT1 to JT8 by superimposing multiple feature points extracted by the feature point position extraction unit 14 onto each of the multiple different transformed processing images, and outputs a feature point distribution screen WD2 containing at least the multiple feature point images to the display device 30. As a result, the image processing device 10 can output feature point images JT1 to JT8, in which the feature points extracted from each of the multiple transformed processing images processed using different parameters are superimposed in a way that allows the user WK1 to compare them.

[0119] Furthermore, the drawing unit 18 outputs a registration recommendation frame WK0 in an identifiable manner around the superior feature point image (for example, feature point image JT7 shown in Figure 9) which has the highest number and proportion of feature points that should be registered based on the shape of the object (for example, part BH) among multiple feature point images. This allows user WK1 to visually determine that feature point image JT7 is the best recommended feature point image to register among multiple feature point images JT1 to JT8, thereby improving the convenience of registering feature points.

[0120] Furthermore, the image processing device 10 is equipped with a communication interface 11 that accepts an operation indicating whether or not to register a parameter as a filter coefficient for matching processing. When the feature selection unit 16 detects a signal from the communication interface 11 indicating that it is to register a parameter as a filter coefficient for matching processing, it registers the specific parameter as a filter coefficient for matching processing in the image filter 13. This allows the user to easily register parameters that they deem appropriate as filter coefficients for matching processing in the image filter 13 through their own operation.

[0121] Furthermore, the drawing unit 18 outputs to the display device 30 a feature point distribution screen WD2 that includes multiple feature point images JT1 to JT8, which are generated based on each of the multiple transformed processing images generated by the image filter 13 using each of the multiple parameters, in a comparative manner. As a result, the image processing device 10 can display each of the multiple feature point images JT1 to JT8 comparatively for viewing by user WK1.

[0122] Although various embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It will be clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these will also be understood to fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention. [Industrial applicability]

[0123] This disclosure is useful as a feature point registration device, feature point registration method, and image processing system that visualize the variation of feature points extracted from each of multiple images of an object and assist in selecting feature points to be registered as a template. [Explanation of Symbols]

[0124] 1 Camera 10 Image Processing Device 11 Communication Interface 12 memory 13 Image Filters 14 Feature Point Location Extraction Unit 15 Feature Calculation Unit 16 Feature Selection Unit 17. Feature Comparison Section 18 Drawing section 20 Operating device 30 Display device 40 Robots 100 Image Processing Systems M1 Image Memory M2 Feature Memory M3 Feature Point Memory

Claims

1. A feature point extraction unit extracts multiple feature points relating to the object from each of several different input images in which the object is shown, A feature comparison unit that compares the feature points extracted from each of the multiple different input images between the input images, An operation reception unit that receives user input regarding feature points, The system includes a registration unit that, based on the comparison results of the feature points, registers some of the feature points extracted from at least one of the input images in association with the object when the operation is accepted, Feature point registration device.

2. The following features are further provided: a feature point selection unit that, in response to the operation, selects feature points that are not to be registered from among the plurality of feature points extracted from at least one input image, The registration unit registers the feature points that were not selected as feature points not to be registered by the feature point selection unit, associating them with the object. The feature point registration device according to claim 1.

3. The system further includes an image filter that applies image transformation processing to the input image using predetermined parameters, The feature point extraction unit extracts the plurality of feature points based on the transformed image, which is the input image after the image transformation process by the image filter. The feature point registration device according to claim 1.

4. The system further comprises a feature calculation unit that calculates the feature quantities of each of the multiple feature points extracted from the input image, The feature comparison unit calculates and compares the similarity of the feature points between the converted images using the calculation results of the feature quantities corresponding to the feature points extracted in the converted image. The feature point registration device according to claim 3.

5. The feature comparison unit, after the number of converted processed images for which the feature quantities corresponding to the feature points have been calculated reaches a predetermined value, calculates and compares the similarity of the feature points among the converted processed images of that predetermined number. The feature point registration device according to claim 4.

6. The system further includes an output unit that generates a feature point image by superimposing the feature points extracted by the feature point extraction unit on one of the aforementioned multiple different input images, and outputs a screen containing at least the feature point image to a display device. The feature point registration device according to claim 1.

7. The output unit outputs the multiple different input images and the feature point images to the display device, including them on the screen in a way that allows for comparison. The feature point registration device according to claim 6.

8. The system further comprises a feature calculation unit that calculates the feature quantities of each of the multiple feature points extracted from the input image, The feature comparison unit calculates and compares the similarity of the feature points between the input images using the calculation results of the feature quantities corresponding to the feature points extracted in the input images. The output unit colors the result of the calculation of the similarity of the feature points and draws the feature points on the feature point image. The feature point registration device according to claim 7.

9. The aforementioned image conversion process is either a smoothing process or a sharpening process. The feature point registration device according to claim 3.

10. The system further comprises an image memory that stores a plurality of different input images received from a camera capable of imaging the aforementioned object, The feature point registration device according to claim 1.

11. The system further includes an input interface that accepts adjustment operations for the predetermined parameters, The image filter applies the image transformation process to the input image using the predetermined parameters after the adjustment operation. The feature point registration device according to claim 3.

12. A feature point registration method performed by a feature point registration device, The process involves inputting multiple different input images showing the object, The steps include extracting multiple feature points relating to the object from each of the multiple different input images, The steps include comparing the feature points extracted from each of the plurality of different input images among the input images, A step to accept user input regarding feature points, Based on the comparison results of the feature points, if the operation is accepted, the system includes the step of registering some of the feature points extracted from at least one of the input images in association with the object. Method for registering feature points.

13. A camera capable of imaging an object, The system includes a feature point registration device that is communicatively connected to the camera, The feature point registration device is A feature point extraction unit extracts multiple feature points relating to the object from each of multiple different input images in which the object is shown. A feature comparison unit that compares the feature points extracted from each of the multiple different input images between the input images, An operation reception unit that receives user input regarding feature points, The system includes a registration unit that, based on the comparison results of the feature points, registers some of the feature points extracted from at least one of the input images in association with the object when the operation is accepted, Image processing system.

Images