Image processing device for recognizing an image of a recognizable object from input data

Abstract

Image processing device (21) that recognizes an image of an object from input data acquired by a vision sensor (10) based on a model pattern comprising a set of several first feature points representing a shape of the object, the device comprising: an object recognition unit (23) that, for each of several input data values ​​obtained by acquiring the object, recognizes an image of the object by matching several second feature points extracted from the input data with several first feature points forming the model pattern; a unit (24) for selecting corresponding second feature points corresponding to the several first feature points forming the model pattern for each of the images of the object recognized from the several input data values, from the several second feature points forming the image.selects and stores the selected second feature points as corresponding points in association with the first feature points; and a model pattern correction unit (25) which, for each of the multiple first feature points forming the model pattern, calculates a statistic of a predetermined physical quantity of the multiple corresponding points associated with the first feature point and corrects the first feature point on the basis of the calculated statistic of the predetermined physical quantity, wherein the unit (24) for selecting corresponding points for each of the at least one recognized image of the object, stores at least one second feature point that does not correspond to any of the first feature points forming the model pattern, stores multiple second feature points forming the image as a feature point candidate, and stores a second feature point corresponding to the at least one stored feature candidate.selects and stores the selected second feature point as a corresponding point in association with the feature point candidate, and the model pattern correction unit (25) further computes for each of the at least one stored feature point candidate a statistic of a predetermined physical quantity of several corresponding points associated with the feature point candidate and adds the feature point candidate to the model pattern as a first feature point based on the statistic of the predetermined physical quantity.

Description

General state of the art The present invention relates to a device that uses feature points forming an image of an object as a model pattern and assesses the degree of similarity between the model pattern and feature points extracted from input data in order to detect the position of the image of the object captured in the input data. When an image of a specific object is recognized from input data captured by a vision sensor using an image processing device, it is common practice to compare feature points between reference information (usually called, for example, a model pattern or template) representing the object and the input data captured by the vision sensor, and to determine that the object has been successfully recognized if the degree of match is higher than a predetermined level. For example, if the vision sensor uses a camera and the input data is a grayscale image, edge points are available as a type of feature point used in such a match. Edge points are points where the luminance changes significantly in a photographic image. Since the contour sections of an object's image generally exhibit significant changes in luminance, edge points are often used as feature points representing the object's contour edge shape. A set of edge points extracted from a photographic image, including a target object, is stored as a model pattern, and the object is recognized based on the degree of similarity between a set of edge points extracted from a photographic image captured by the vision sensor and the set of edge points that make up the model pattern. The generalized Hough transform, etc., is then used to perform this process.are known as techniques of this type. The shapes of the contour edges of images of an object captured in photographs vary. This is due, for example, to: the individual differences in the object's shape; and the large parallax. Parallax, in this context, refers to the deviation in the positional relationship of contour edges captured in a photographic image, due to changes in the relative position between an object and a vision sensor, when the object is three-dimensional or has rounded edges. The goal is to enable precise object recognition, even when the object's contour edges exhibit significant variations. However, in practice, it is often the case that an individual object used to teach a model pattern, or even its photographic image, deviates due to variation and is therefore not optimal for matching. In such a case, the expected recognition is impossible. For example, recognition may be possible for one type of variation, even if the variation is very large, while recognition for another type of variation is impossible, even if the variation is minor. Under these circumstances, with regard to applying the aforementioned technique, an optimal model pattern can preferably be generated according to the variations that are expected to occur. In JP 5 080 416 B2, when an object is detected that matches a model of an input photograph, the degree of contribution of each feature set of the model to the result is accumulated during the matching process. A method for removing unwanted feature sets based on the accumulated degree of contribution is disclosed. The method disclosed in JP 5 080 416 B2 is used to correct a model of information regarding the match at the time of actual detection; however, it can undesirably remove a model feature set in a section that exhibits large individual variations in the shape of a target object, because that section has a low degree of contribution. Furthermore, this method may not correct the shape of a model pattern to the optimal shape for a match. JP 2015-007972A discloses a method for generating multiple varying photographic images by applying various variations, such as rotation, enlargement or reduction, distortion, changes in lighting, and image noise, to a model photograph. Feature sets of the pattern are extracted from the varying photographic images to generate a feature set distribution. This distribution can be used to generate a more robust model. In the method disclosed in JP 2015 - 007 972 A, variations due to factors associated with the actual individual differences or environments of an object to be detected cannot be reflected on the model, since variations are applied using an artificial method. JP 4 843 415 B2 discloses a method for repeatedly capturing the same object from the same perspective, performing a detection for each of several photographic images, and calculating the average position of detected contour edges to improve the accuracy of the detected position. This method is used to reduce the influence of measurement errors of the same object and consequently cannot reduce the influence of variations in multiple objects to be detected. JP H07-037085A discloses a method for cutting a region matching an initial reference pattern from a training photograph as a partial photograph and averaging several cut partial photographs for each pixel value to generate a new reference pattern. Averaging for each pixel can remove variations due to noise. However, if the positions of edges exhibit variations due to the influence of individual differences or parallax, the luminance gradient in the average photograph may be small in this method, leading to inaccurate edge extraction. US Patent 2015 / 0 235 380 A1 discloses a device for detecting three-dimensional objects. The device comprises an imaging unit configured to capture images of a search area and generate image data; a fitting unit configured to compare a three-dimensional object in an image with a three-dimensional shape model corresponding to the three-dimensional object, based on the image data, in order to identify correlating feature points by pattern matching; and a model update unit configured to update the three-dimensional shape model based on the feature points identified by the fitting unit.The device further comprises a motion estimation unit configured to estimate movement of the three-dimensional object based on a history of the position and orientation of the three-dimensional shape model, a validity determination unit configured to compare the feature points determined by the fitting unit with the three-dimensional shape model estimated by the motion estimation unit, and to cause the model update unit to update the three-dimensional shape model only based on valid feature points. US Patent 2004 / 0131247A1 discloses an image processing device comprising an image drawing section, an image input section, a correspondence point search section, and a processing parameter enhancement section. The image drawing section has a preset three-dimensional model image to draw a three-dimensional image based on the three-dimensional model image and a first processing parameter. An input image captured by a camera is entered into the image input section. The correspondence point search section looks for the point on the input image that corresponds to a specific point on the three-dimensional image drawn by the image drawing section in order to obtain information about the position of the correspondence point.The section for improving processing parameters uses the information obtained from the section for finding correspondence points regarding the position of the correspondence point to improve a value for the first processing parameter. US Patent 2015 / 0009214A1 discloses methods and systems for generating a three-dimensional (3D) model of a fully formed object depicted in a noisy or incomplete scene. An image processing module of a computer device receives images captured by a sensor. Based on an analysis of the images, the module generates partial 3D mesh models of physical objects in the scene and determines the location of at least one target object in the scene by comparing the images with one or more 3D reference models and extracting a 3D point cloud of the target object. The module compares the 3D point cloud of the target object with a selected 3D reference model using a similarity parameter and captures one or more features of the target object.The module generates a fully formed 3D model of the target object using partial or noisy 3D points from the scene, extracts the detected features of the target object and features of the 3D reference models that correspond to the detected features, and calculates measurements of the detected features. The document “An On-line Visual Human Tracking Algorithm Using SURF-based Dynamic Object Model,” by A. Meenakshi Gupta et al., published in Transactions on the IEEE 2013 International Conference on Image Processing, pages 3875 to 3879, discloses an interest-point-based tracking algorithm for non-stationary objects, such as people in a non-stationary video. The tracking algorithm utilizes a dynamic object model that evolves over time to account for changes that may occur due to pose changes in successive frames. This dynamic object model aims to obtain a set of key points necessary for tracking the target. An autoregression model is used to predict the target's location if the target becomes obscured. WO 2015 / 107 859 A1 discloses an image comparison device comprising: a storage unit that stores a model image containing the locations of feature points of an object to be compared and the directions of the brightness gradients at each feature point location; a feature value acquisition unit that acquires the feature point locations and the directions of the brightness gradients at each feature point location; and an image comparison unit that performs a comparison between the model image and the input image. The image comparison unit defines permissible ranges for both the feature point locations and the directions of the brightness gradients at each feature point location and performs the image comparison. Brief description of the invention It is an object of the invention to enable precise recognition of an object, even if the shapes of the contour edges of images of the object captured in photographic images vary. According to the disclosure, an image processing device is provided in accordance with the independent claim. Developments are described in the dependent claims. Brief description of the drawings The present invention will be more clearly understood by reference to the following accompanying drawings, in which: Fig. 1 is a block diagram showing the configuration of an image processing device mounted in a vision sensor controller; Fig. 2 is a diagram showing an exemplary configuration when the position of an object is detected by a vision sensor and the vision sensor controller, which includes the image processing device according to one embodiment; Fig.Figure 3 is a diagram illustrating an exemplary configuration in which, when an object is handled on a worktable by the hand of a robot controlled by a robot controller, a photographic image captured by the vision sensor located on the end effector of the robot is processed by the image processing device according to the embodiment, which is mounted in the vision sensor controller, to recognize the object, and the position information of the recognized object is provided to the robot controller; Figure 4 is a flowchart illustrating processing according to a first embodiment; Figure 5 is a flowchart illustrating a process for generating a model pattern by an image processing unit; Figure 6 is a diagram illustrating an exemplary method for calculating the distance between the position of a first feature point and that of a second feature point; Figure7 is a flowchart illustrating processing according to a second embodiment; Fig. 8 is a diagram illustrating processing for selecting a feature point candidate; Fig. 9 is a diagram illustrating processing for removing duplicate feature point candidates; Fig. 10 is a flowchart illustrating processing according to a third embodiment; Fig. 11 is a diagram illustrating an exemplary model pattern of the object; Fig. 12 is a view illustrating an exemplary model pattern marker region in a captured photographic image; Fig. 13 is a table illustrating the data format of corresponding points stored in a model pattern storage unit; Fig. 14 is a diagram illustrating a method for calculating an error vector; Fig.Figure 15 is a diagram illustrating a method for correcting first feature points in the model pattern based on statistics; Figure 16 is a table showing the format of determination indices stored in the model pattern storage unit; Figure 17 is a table showing the data format of first feature points stored in the model pattern storage unit; Figure 18 is a table showing the data storage format of error vectors; and Figure 19 is a flowchart showing processing according to a fourth embodiment. Detailed description The first to third embodiments (which in a description of common parts herein are sometimes simply referred to as an "embodiment") of the present invention are described below with reference to the drawings. It should be understood, however, that the present invention cannot be limited to either the drawings or the following embodiments. A description of an embodiment uses the following symbols. For the sake of clarity, the symbols are described first. NP: the number of first feature points that form a model pattern. P_i: the i-th first feature point (i = 1 to NP) of the model pattern. NI: the number of input images. I_j: the j-th input image (j = 1 to NI). NQ_j: the number of second feature points extracted from the j-th input image I_j. Q_jk: the k-th second feature point (k = 1 to NQ_j) extracted from the j-th input image I_j. NT_j: the number of images of an object recognized from the j-th input image I_j. T_jg: the g-th image (g = 1 to NT_j) of the object recognized from the j-th input image I_j. NT: the total number of images of the object recognized from all input images (NT = ΣNT_j). NO_jg: the number of feature points that form the image T_jg of the object, the second feature points Q_jk,that are extracted from the j-th input image I_j (NO_jg < NQ_j)O_jgn: the n-th feature point (non-corresponding point) (n = 1 to NO_jg) of feature points that are not the feature points forming the image T_jg of the object, of the second feature points Q_jk extracted from the j-th input image I_jNO_i: the number of corresponding points determined to correspond to the i-th feature point P_i of the model pattern (NO_i ≤ NT)O_im: the m-th corresponding point (m = 1 to NO_i) that corresponds to the i-th feature point P_i of the model patternNC: the number of feature point candidatesC_h: the h-th feature point candidate (h = 1 to NC)NO_h: the number of corresponding points determined to correspond to feature point candidate C_h (NO_n < NT)O_hp: the p-th corresponding point (p = 1 to NO_h) that corresponds to the feature point candidate C_h, Fig. 2 is a diagram showing an exemplary configuration when the position of an object is detected by a vision sensor and a vision sensor controller comprising an image processing device according to one embodiment. A vision sensor 10 is fixed in a position where it can detect an object 1, and the object 1 is placed on a worktable 2, as shown in Fig. 2. The vision sensor 10 can be implemented in either a camera that captures a grayscale or color image, or a stereo camera or a three-dimensional sensor that can obtain an area image or a set of three-dimensional points. A set of three-dimensional points can be obtained on either the contour edges of the object or the plane of the object. This embodiment requires the use of a camera, designated as the vision sensor 10, which outputs a grayscale image. The camera is implemented in a digital camera comprising an image sensor, such as a CCD (charge-coupled device), which serves as a well-known light-receiving device with the function of detecting a two-dimensional photographic image on its image acquisition plane (on its CCD array plane) by capturing an image. A two-dimensional coordinate system on the image acquisition plane is hereinafter referred to as a photographic image coordinate system. Fig. 3 is a diagram illustrating an exemplary configuration in which, when the object 1 is handled on the worktable 2 by a hand 12 of a robot 11 controlled by a robot controller 13, a photographic image captured by the vision sensor 10, located on the end effector of the robot 11, is processed by the image processing device according to the embodiment, which is mounted in a vision sensor controller 20, to recognize the object 1, and the position information of the recognized object 1 is provided to the robot controller 13. The vision sensor 10 can be located on a moving section, such as the end effector of the robot 11, as shown in Fig. 3. Fig. 1 is a block diagram showing the configuration of the image processing device mounted in the vision sensor controller. An image processing device 21 comprises an image processing unit 22, a model pattern storage unit 26, and a recognition result storage unit 27, as shown in Fig. 1. The image processing device 21 is connected to, for example, a vision sensor 10, a control panel 31, and a display 32. The image processing unit 22 comprises an object recognition unit 23, a unit 24 for selecting corresponding points, and a model pattern correction unit 25. The model pattern storage unit 26 stores a learned model pattern. The recognition result storage unit 27 stores the result of object recognition from input data using the learned model pattern. Each part included in the image processing device 21 is implemented as software on a computer, which includes, for example, a CPU, a ROM, and a RAM.The first to the third embodiment (which will be described later) differ from each other in terms of the details of the processing, which is implemented as software. The vision sensor 10 is connected to the image processing device 21 via a communication cable. The vision sensor 10 provides captured image data to the image processing device 21. The control panel 31 is also connected to the image processing device 21 via a communication cable. The control panel 31 is used to configure the vision sensor 10, which is preferably involved in the object 1 detection by the image processing device 21. The display 32 is connected to the image processing device 21 via a communication cable. The display 32 shows an image captured by the vision sensor 10 and details the settings on the control panel 31. The vision sensor 10, the control panel 31, and the display 32 can be integrated with the image processing device 21. Fig. 4 is a flowchart illustrating processing in the image processing device 21 according to a first embodiment. The details of a processing operation carried out by the image processing unit 22 in the first embodiment are described below with reference to Fig. 4. In step S101, the image processing unit 22 generates a model pattern and stores the generated model pattern in the model pattern storage unit 26. The model pattern in this embodiment is formed by several feature points. Although various points are available as feature points, boundary points are used as feature points in this embodiment. Boundary points exhibit high luminance gradients in a photographic image and can be used to obtain the contour shape of object 1. Since a method for extracting boundary points is described, for example, in "Computer Vision," Kyoritsu Shuppan Co., Ltd., January 2007, and is well known, a description of this method is not included here. Examples of the physical quantities of an edge point include the position, the luminance gradient direction, and the luminance gradient magnitude of the edge point. After defining the luminance gradient direction of an edge point as the orientation of a feature point, the orientation can be combined with the position to define the position / orientation of the feature point. In this embodiment, the physical quantities of a feature point are the physical quantities of an edge point, i.e., the position, the orientation (luminance gradient direction), and the luminance gradient magnitude of the edge point. Fig. 11 is a diagram representing an exemplary model pattern of object 1. The model pattern of object 1 is formed by several first feature points P_i (i = 1 to NP), as shown in Fig. 11. The position / orientation of the first feature point P_i that forms the model pattern can be represented in any form, examples of which include a method for defining a coordinate system 100 (hereinafter referred to as a model pattern coordinate system 100) for the model pattern and representing a position t_Pi (i = 1 to NP) and an orientation v_Pi (i = 1 to NP) of a feature point that forms the model pattern, using, for example, a position vector and a direction vector when considering the model pattern coordinate system 100. The starting point of the model pattern coordinate system 100 can be defined in any way. For example, an arbitrary point can be selected from the first feature points that form the model pattern and defined as the starting point, or the barycenter of all feature points that form the model pattern can be defined as the starting point. The orientation (axis direction) of the model pattern coordinate system 100 can also be defined in any way. For example, the orientation can be defined such that the model pattern coordinate system 100 is parallel to a photographic image coordinate system in a photograph that was used to create a model pattern, or two arbitrary points can be selected from the feature points that form the model pattern and can be defined such that the direction from one to the other of these two points coincides with the X-axis direction. The first feature points P_i, which form the model pattern, are stored in the model pattern storage unit 26 in a form (which includes the position, orientation and luminance gradient magnitude), as shown, for example, in Fig. 17. Fig. 5 is a flowchart illustrating a process for generating a model pattern by the image processing unit 22. In step S201, an object 1, which is to be taught as a model pattern, is located within the field of view of the vision sensor (the camera) 10 in order to capture a photograph of the object 1. The positional relationship between the camera 10 and the object 1 at this time is desirable to be the same as when the object 1 was detected. In the captured photographic image in step S202, the region capturing object 1 is marked as a model pattern marker region using a rectangle or a circle. Fig. 12 is a view showing an exemplary model pattern marking region in a captured photographic image. Referring to Fig. 12, a photo image coordinate system 210 is defined in a captured photo image, and a model pattern marker region (in this case, a rectangular region) 220 is marked to encompass an image 1A of the object 1 in the photo image coordinate system 210. The model pattern marker region 220 can be set by the image processing unit 22 in response to a user instruction via the control panel 31 while browsing a photo image on the display 32, or a section with a high luminance gradient in the captured photo image can be obtained by the image processing unit 22 as the outline of an image 1A, so that the model pattern marker region 220 is automatically marked to encompass the image 1A. In step S203, boundary points are extracted as feature points within the area of ​​the model pattern marking region 220 to obtain physical quantities, such as the positions, orientations (luminance gradient directions), and luminance gradient magnitudes, of the boundary points. A model pattern coordinate system 100 is defined in the marked region, and the positions and orientations of the boundary points are converted from values ​​represented in the photographic image coordinate system 210 to values ​​represented in the model pattern coordinate system 100. In step S204, the physical quantities of the extracted boundary points are stored in the model pattern storage unit 26 as the first feature points P_i that form the model pattern. Although boundary points are used as feature points in this embodiment, the feature points applicable in this embodiment are not limited to boundary points, and feature points such as SIFT can be used. A method for extracting SIFT feature points from photographic images is described, for example, in David G. Lowe, “Object Recognition from Local Scale-Invariant Features,” Proc. of the International Conference on Computer Vision, Corfu (Sept. 1999), and is well known. Instead of extracting, for example, boundary points or SIFT feature points from a photograph of object 1 and setting these as the first feature points forming the model pattern, geometries such as segments, rectangles, or circles can be arranged in accordance with the contour edges of the object captured in a photograph to generate a model pattern. In this case, feature points can preferably be arranged with adequate spaces between them on the geometry forming the contour edges. A model pattern can also be generated based on, for example, CAD data. For two-dimensional CAD data, a model pattern can be generated using the same method as that which uses geometries. For three-dimensional CAD data, it may be preferable to project the shape of an object represented by the CAD data onto a photographic image, extract feature points from the projected image, and set these as the initial feature points. This conversion can be performed, for example, by the following steps: 1. A local coordinate system is defined with its starting point on the image acquisition plane of a camera. 2. The camera is pre-calibrated. This can convert three-dimensional points represented in the local coordinate system into two-dimensional points in a camera photograph. 3. An object represented as CAD data in the local coordinate system is virtually placed.The placed CAD data is displayed in the local coordinate system. The relative relationship between the camera and the object is set approximately the same as that used when actually detecting the object. 4. A set of three-dimensional points on contour edges is obtained from the contour edges at a predetermined interval. Contour edges, used as a model pattern, are marked from the CAD data as appropriate. 5. The set of three-dimensional points is projected onto the camera image to obtain a set of two-dimensional points in the image coordinate system. Marking the direction of a light-dark pattern on the CAD data allows for the addition of the luminance gradient direction. The direction of a light-dark pattern refers to information indicating which of two regions, divided by a contour edge as a boundary, is lighter. 6.The resulting set of two-dimensional points in the photographic image coordinate system is converted to be represented in a model pattern coordinate system and is stored in the model pattern storage unit as the first feature points. Referring again to step S102, one or more input photo images I_j (j = 1 to NI) comprising image 1A of object 1 are provided. The image 1A of object 1 captured in the input photo image I_j ideally includes variations of interest for recognition. For example, if the individual differences of object 1 are of interest, preferably multiple objects 1 with individual differences are provided to use captured images of objects 1 as input photo images. If the difference in view, which depends on the position where object 1 is placed, is of interest, preferably photo images obtained by capturing the object as if placed in different positions / orientations within the camera's field of view are used as input photo images.A large number of photographs of object 1, actually captured on a production line, can be stored and used as input images. Photographs actually captured on a production line encompass a variety of variations that need to be considered and are therefore suitable. Multiple objects can be captured in a single input image. In step S103, image 1A of object 1 (hereinafter sometimes simply referred to as object 1) is recognized for each input photo image I_j (j = 1 to NI). Second feature points are first extracted from the input photo image I_j. Second feature points can preferably be extracted using the same method as that used to extract first feature points when generating a model pattern. In this embodiment, boundary points are extracted from the input photo image and set as second feature points. For convenience, the second feature points extracted from the input photo image I_j are defined as Q_jk (k = 1 to NQ_j). The second feature points Q_jk are stored in the recognition result storage unit 27 in association with the input photo image I_j. At this point, the positions of the second feature points Q_jk are represented in the photo image coordinate system 210. Object 1 is recognized by comparing the second feature points Q_jk, which were extracted from the input photo image I_j, and the first feature points P_i, which form the model pattern. Although various methods are available for object recognition, examples of available, well-known methods include the generalized Hough transform, described in DH Ballard, "Generalizing the Hough Transform to Detect Arbitrary Shapes," Pattern Recognition Vol. 13, No. 2, pp. 111-122, 1981; RANSAC, described in Martin A. Fischler et al., "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the Association for Computing Machinery Vol. 24, No. 6, pp. 381-395, March 1980; and the ICP algorithm, described in Paul J. Best et al., "A method for Registration for 3-D Shapes," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No.2 February 1992 , is described. As a result of the recognition, NT_j images of the object were recognized from the input photo image I_j. Let T_jg (g = 1 to NT_j) be the recognized image and R_Tjg be the recognition position of the image T_jg. The recognition position R_Tjg is a homogeneous transformation matrix that represents the position of the image T_jg of the object when viewed from the photo image coordinate system 210, i.e., the position of the model pattern coordinate system 100 when viewed from the photo image coordinate system 210 at the time of superimposition of the model pattern onto the image T_jg, and is given by the following: For example, if the object is not tilted with respect to the optical axis of the camera and it is sufficient to consider only a congruent transformation as the movement of an image of the object captured in a photograph, then a00 to a12 are given by the following: where (x, y) is the position on the photograph and e is the extent of the rotation on the photograph. If the object is not tilted with respect to the camera's optical axis, and the distance between the object and the camera is not constant, the size of an image of the object captured in a photograph changes depending on the distance, and a similarity transformation is applicable as a motion of an image of the object captured in a photograph. In this case, a00 to a12 are given by the following: where s is the ratio between the size of the taught model pattern and that of the image T_jg of the object. The same processing is performed for each input image I_j (j = 1 to NI), and a sum of NT images is detected. The total number of NT is given by the following: The recognition positions R_Tjg are stored in the recognition result storage unit 27 in association with the input photo image I_j. In step S104, second feature points Q_jk, corresponding to the first feature points P_i that form the model pattern, are selected from the second feature points Q_jk (j = 1 to NI, k = 1 to NQ_j) extracted from the input photo image I_j, as corresponding points based on the recognition position R_Tjg of the image T_jg (j = 1 to NI, g = 1 to NT_j) of the object that was recognized from each input photo image I_j (j = 1 to NI). For convenience, the position of each first feature point P_i that forms the model pattern is represented by a homogeneous transformation matrix R_Pi. R_Pi can be expressed as follows: where t_Pi = (tx_Pi, ty_Pi) is the position of P_i in the model pattern coordinate system and v_Pi (vx_Pi, vy_Pi) is the position of P_i in the model pattern coordinate system. The position of P_i can also be represented by an angle r_Pi instead of a vector. v_Pi can be represented using r_Pi as v_Pi (vx_Pi, vy_Pi) = (cos r_Pi, sin r_Pi). Similarly, the position of every second feature point Q_jk, which was extracted from the input photo image I_j, is also represented by a homogeneous transformation matrix R_Qjk. Note that the position R_Pi of the first feature point P_i, which forms the model pattern, is represented in a model pattern coordinate system, and the position R_Qjk of the second feature point Q_jk, which was extracted from the input photo image I_j, is represented in a photo image coordinate system. The relationship between these two coordinate systems is clearly defined. If R_Pi' is defined as the position of the first feature point P_i when viewed from the photographic coordinate system at the time the model pattern is superimposed onto the image T_jg of the object captured in the photograph I_j, then R_Pi' can be expressed using the position R_Pi of the first feature point P_i when viewed from the model pattern coordinate system and the recognition position R_Tjg of the image T_jg when viewed from the photographic coordinate system as follows: Similarly, if R_Qjk' is defined as the position of the second feature point Q_jk when viewed from the model pattern coordinate system at the time the model pattern is superimposed onto the image T_jg of the object, then R_Qjk' can be expressed using the position R_Qjk of Q_jk when viewed from the photo image coordinate system and the recognition position R_Tjg of the image T_jg when viewed from the photo image coordinate system as follows: For the sake of later convenience, let t_Pi' be the position of P_i when viewed from the photographic coordinate system, v_Pi' be the position of P_i when viewed from the photographic coordinate system, t_Qjk be the position of Q_jk when viewed from the photographic coordinate system, v_Qjk be the position of Q_jk when viewed from the photographic coordinate system, T_Qjk' be the position of Q_jk when viewed from the model pattern coordinate system, and v_Qjk' be the position of Q_jk when viewed from the model pattern coordinate system. In light of the above description, the first feature points P_i, which form the model pattern, and the second feature points Q_jk (j = 1 to NI, k = 1 to NQ_j), extracted from the input photo image I_j, are associated according to the following procedure: 1. The position R_Pi of the first feature point P_i, which forms the model pattern, is transformed into a position R_Pi' when viewed from the photo image coordinate system according to equation (1) based on the recognition position R_Tjg of the image T_jg of the object recognized from the input photo image I_j. 2. For each first feature point P_i, a search is conducted for one of the second feature points Q_jk that is closest to the first feature point P_i. The following procedure can be used for this search.(a) The distances between the position R_Pi' of the first feature point and the positions R_Qjk of all second feature points are calculated, and a second feature point Q_jk that is closest to the first feature point is selected. (b) The position R_Qjk of the second feature point is stored in an element corresponding to a pixel at that position in a two-dimensional array containing elements equal to the number of pixels in the input image I_j. The array is then searched two-dimensionally for the nearest pixel corresponding to the unique position R_Pi of the first feature point, and the first second feature point Q_jk found is selected. 3. It is evaluated whether the selected second feature point Q_jk is an adequate corresponding point for the first feature point P_i.For example, the distance between the position R_Pi' of the first feature point P_i and the position R_Qjk of the second feature point Q_jk is calculated, and the selected second feature point Q_jk is determined to be an adequate corresponding point for the first feature point P_i if the obtained distance is less than or equal to a limit value. The differences in physical quantities, such as position and luminance gradient magnitude, between the first feature point P_i and the second feature point Q_jk are evaluated together, and a second feature point Q_jk, selected if these differences are also less than or equal to or greater than or equal to limits, can be determined to be an adequate corresponding point for the first feature point P_i. 4.If the selected second feature point Q_jk is determined to be an adequate corresponding point for the first feature point P_i, this second feature point Q_jk is determined to be a corresponding point O_im for the first feature point P_i and stored in the recognition result storage unit 27 in association with P_i. A position R_Oim of the corresponding point O_im when viewed from the photo image coordinate system satisfies R_Oim = R_Qjk, is the position when viewed from the photo image coordinate system, and is consequently converted into a position R_Oim' when viewed from the model pattern coordinate system, and the position and orientation of R_Oim' are stored. From equation (2), R_Oim' can be calculated as follows: The distance between the position t_Pi' of the first feature point P_i and the position t_Qjk of the second feature point Q_jk can be determined using a direct distance between two points, but can also be determined using a value calculated using the following procedure shown in Fig. 6: 1. A straight line passing through the position t_Pi' of the first feature point P_i and parallel to the position v_Pi' of the first feature point P_i is defined as L2. 2. A straight line passing through the position t_Qjk of the second feature point Q_jk and perpendicular to the position v_Qjk of the second feature point v_Qjk is defined as L1. 3. A distance d_gi between a node T between L1 and L2 and the position t_Pi' of the first feature point P_i is calculated. In this case, the distance d_gi is signed. When the distance is calculated in this way, a deviation in a direction perpendicular to the luminance gradient direction of the first feature point P_i is ignored. After the processing described above, for each of NT recognition positions R_Tjg (j = 1 to NI, g = 1 to NQ_j) recognized from the input image frames I_j (j = 1 to NI), NO_i corresponding points were found, which were determined to correspond to the i-th first feature point P_i of the model pattern. Let O_im (m = 1 to NO_i) be the m-th corresponding point corresponding to the i-th first feature point P_i of the model pattern. Since the total number of images of the object recognized from the input image frames I_j is NT, NO_i ≤ NT. The obtained corresponding points are stored in the model pattern storage unit 26 in a form as shown in Fig. 13. Although in this embodiment appropriate points are selected for all NT detection positions R_Tjg (j = 1 to NI, g = 1 to NQ_i), this processing cannot always be performed for all detection positions. For example, if clearly inadequate detection positions are mixed, they should ideally be excluded. Exclusion can be performed using, for example, the following method. As an exemplary elimination procedure, the luminance values ​​in the vicinity of a detection position are compared at the time of model pattern teaching and at the time of detection. The detection position is determined to be inadequate and is eliminated if these luminance values ​​differ significantly. This elimination process is described below. 1. A region at which a luminance value is measured is set. This region is referred to hereafter as a measurement region. A measurement region can be located either inside or outside the model pattern. Although no specific restriction is imposed on the shape of a measurement region, it is assumed that, in this case, the measurement region forms a rectangle, and the position and size of the rectangle when viewed from the model pattern coordinate system are stored.Furthermore, in a photograph obtained by teaching a model pattern, the statistics of the luminance value in the measurement region are calculated and stored. Possible examples of statistics include average, variance, maximum, and minimum values ​​of the luminance value. 2. A position R_C' of the measurement region, corresponding to the recognition position R_Tjg of the recognition result T_jg, is calculated. If a position / position R_C of the rectangle of the measurement region is defined, R_C' can be calculated based on R_C and R_Tjg as follows: 3. The statistics of the luminance value of the input photograph I_j in the rectangular region defined by R_C' are calculated. 4. If the difference between the statistics of the luminance value in the measurement region of the model pattern and those of the luminance value in the measurement region of the recognition result T_jg is greater than or equal to a threshold value, this recognition result is determined to be inadequate and is rejected. The above-mentioned processes in 2 to 4 are performed for each recognition result T_jg (g = 1 to NT_j). The physical quantity used to determine exclusion or non-exclusion is not limited to the luminance value. For example, the magnitude of the luminance gradient in the measurement region, or the hue or saturation, can be used. As long as the statistics of a physical quantity that can have such an effect are calculated, the present invention is, of course, not limited to the examples given herein. As another exclusion procedure, the user can manually exclude an inadequate recognition result. A recognition result is displayed on screen 32 and visually inspected by the user to avoid using a recognition result that has been determined to be inadequate. The processing mentioned above (a check for the adequacy of a recognition result) is performed for each recognition result T_jg (g = 1 to NT_j). Excluding inadequate detection results can prevent such detection results from adversely affecting the calculation of statistics in the subsequent step. In step S105, the statistics of the physical quantity of corresponding points that correspond to the first feature point P_i are calculated on the basis of the physical quantities at NO_i corresponding points O_im (m = 1 to NO_i), which were determined to correspond to the i-th first feature point P_i that forms the model pattern. A procedure for obtaining an average error vector of the position of a feature point as an exemplary statistic is described below. (a) Positions R_Oim' of NO_i corresponding points O_im, which correspond to the first feature point P_i, when viewed from the model pattern coordinate system, which are stored in the recognition result storage unit 27, are obtained. (b) As shown in FIG. 1, an error vector V_im is then calculated for each of the NO_i corresponding points O_im by subtracting the position component t_Pi of the position R_Pi of the first feature point P_i from the position component t_Oim' of the position R_Oim' of the corresponding point O_im. NO_i error vectors V_im are thus obtained. The obtained error vectors are stored in the model pattern storage unit 26 in a form as shown in FIG. 18.(c) All of the NO_i error vectors V_im are summed and the sum is divided by NO_i to compute an average error vector V_i, which is stored in the model pattern storage unit 26 in association with the feature point P_i as the statistic of the first feature point P_i. V_i is given by the following: Although all error vectors V_im for the first feature point P_i are stored in the model pattern storage unit and then their average is calculated in the example mentioned above, an average error vector V_i can be obtained by sequentially adding error vectors V_im and finally dividing their sum by NO_i. An average error vector can also be obtained using a different method. For example, a probability distribution with NO_i error vectors as the probability variable can be obtained to determine a vector in a section with the highest probability as the average error vector. Since the photographic image is two-dimensional, an average error vector can be calculated by estimating a two-dimensional probability distribution. Examples of probability distributions include a normal distribution and a mixture of normal distributions. Outliers can be removed before calculating an average error vector. Removing outliers can prevent incorrect corresponding points from degrading the positional accuracy of an average error vector. Outliers can be removed using, for example, the following procedure. A probability distribution with an error vector as a probability variable is first obtained. The probability of each error vector being taken is calculated for the obtained probability distribution. Error vectors showing such obtained probabilities lower than a certain threshold are removed as outliers. After the outliers have been removed, an average error vector can preferably be obtained by, for example, obtaining a new probability distribution. The processing mentioned above is performed for all first feature points P_i that form the model pattern. In step S106, the first feature points P_i of the model pattern are corrected based on the obtained statistics (average error vectors V_i). In particular, the position t_Pi of feature point P_i can preferably be changed to the sum t_Pi + V_i of the average error vector V_i added to t_Pi, as shown in Fig. 15. Thus, the shape formed by the feature points P_i (i = 1 to NP) of the model pattern becomes the average shape of NT images T_jg (j = 1 to NI, g = 1 to NT_j) captured in the photographic images I_j (j = 1 to NI). The type of statistic calculated in step S105 and the method of correcting the model pattern in step S106 are not limited to the procedure mentioned above. For example, the following procedure can be used: 1. A distance d_im between the first feature point P_i and the corresponding point O_im is calculated using the procedure shown in Fig. 6. d_im represents a signed distance. Since NO_i corresponding points exist for the first feature point P_i, NO_i distances d_im are calculated for each first feature point P_i. 2. An average d_i of the NO_i distances d_im is calculated according to the following equation. The resulting average distance d_i is stored in the model pattern storage unit in association with the feature point P_i as a statistic. The average d_i is given by the following: 3.The position v_Pi of the first feature point P_i is multiplied by d_i to obtain an average error vector V_i as follows:. The calculated average error vector V_i is added to the position t_Pi of the feature point P_i. The position t_Pi of the feature point P_i is only corrected in the luminance gradient direction. Just as the position t_Pi of the first feature point P_i of the model pattern can be corrected, so too can the position v_Pi of the feature point P_i. An average vector uv_Oi of the position vector v_Oi of the corresponding point O_im of the first feature point P_i can be obtained as uv_Oi = (Σv_Oim) / |Σv_Oim| and the position vector v_Pi of the first feature point P_i can be replaced by uv_Oi. Alternatively, the position vector of the feature point P_i can be transformed into an angle r_Pi = arctan(vx_Pi / vy_Pi), and the angle of the position of the corresponding point O_im, obtained through a similar transformation, can be defined as r_Oim' to obtain a difference of the angle r_im as r_Oim' - r_Pi. Note that r_im is transformed to take on values ​​from -π to π. The position of the feature point P_i can be corrected by adding a difference average = r_i (Σr_im) / (NO_i) to the angle r_Pi of the feature point's position. The following advantageous effects can be obtained by correcting the model pattern in this way. 1. Since a model pattern generated from a photographic image obtained by capturing a specific object reflects the shape of a particular individual of the object, it is not optimal for all individuals. Using this method allows the average shape of the object to serve as a model pattern. The perception of the object can vary depending not only on the differences in shape within each individual of the object, but also on differences in camera position or the object's location. The model pattern can be averaged to include such variations in perception. 2. A model pattern generated from CAD data representing the shape of an object has an ideal shape of the object, but due to manufacturing errors, it is inherently not produced according to its measurements.Furthermore, errors can occur when converting a model pattern represented in a coordinate system from CAD data into a photographic coordinate system. Such a model pattern can be optimized to match the shape of an actual object to be recognized. The same applies when the contour boundaries of a model pattern are directly determined by combining geometries such as segments, rectangles, or circles. A first embodiment has been described above. A second embodiment will be described next. In the second embodiment, even for second feature points that do not correspond to the first feature points of the model pattern, the statistics of the physical quantities of these corresponding points are obtained, and new first feature points are added to the model pattern based on the obtained statistics. Fig. 7 is a flowchart illustrating processing according to the second embodiment. Steps S301 to S303 are identical to steps S101 to S103 in the first embodiment. Steps S301 to S303 recognize NT_j images of the object from an input photo image I_j, as at the end of step S103. The recognition position of a recognized image T_jg (g = 1 to NT_j) is defined as R_Tjg. In step S304, a unit 24 selects corresponding points that correspond to the first feature points forming the model pattern. Although the process in step S304 is essentially the same as in step S104 of the first embodiment, they differ in that the former performs the following additional steps. As shown in Fig. 8, based on the position / orientation R_Tjg of the object's image T_jg, which was recognized from the input image I_j, second feature points Q_jk (points surrounded by a dashed line in Fig. 8) are extracted from the input image I_j, but are not those determined to correspond to the first feature points P_i (i = 1 to NP) that form the model pattern. These second feature points are stored in a recognition result storage unit 27 in association with the object's image T_jg as non-corresponding points O_jgn (n = 1 to NO_jg). Note that NO_jg is a value less than or equal to NQ_j. In step S305, unit 24 selects feature point candidates for the selection of corresponding points and stores them in a model pattern storage unit 26 according to the following procedure. The position R_Ojgn' of the non-corresponding point O_jgn when viewed from the model pattern coordinate system is calculated according to equation (2) mentioned above, based on the position R_Ojgn of O_jgn when viewed from the photographic image coordinate system and the position R_Tjg of the object's image T_jg for each non-corresponding point O_jgn (n = 1 to NO_jg) of the image T_jg stored in the recognition result storage unit 27. It is checked whether R_Ojgn' falls within a model pattern marker region, and O_jgn is excluded from the non-corresponding points of the object's image T_jg if R_Ojgn' falls outside this region. The model pattern marker region used in this case can be a region other than the one used to generate a model pattern. The remaining non-matching points are stored in association with the image T_jg of the object as feature point candidates C_h. When adding a feature point candidate, the physical size of the non-matching point O_jgn can be used to further narrow down the candidates. For example, no non-matching points with a luminance gradient magnitude smaller than a certain threshold can be added as feature point candidates. The operations in steps S304 and S305 are performed for each image T_jg (j = 1 to NI, g = 1 to NT_j) of the object recognized from each input image I_j (j = 1 to NI). Initially, there are zero feature point candidates C_h, but one feature point candidate C_h can be added each time the operations in steps S304 and S305 are performed, and eventually NC feature point candidates C_h (h = 1 to NC) are obtained. In step S306, corresponding points O_hp (p = 1 to NO_h), which correspond to the feature point candidates C_h (h = 1 to NC), are selected from the second feature points Q_jk (j = 1 to NI, k = 1 to NQ_j), which were extracted from the input photo image I_j, on the basis of the position R_Tjg (j = 1 to NI, g = 1 to NT_j) of the image of the object. Although this process is essentially the same as in step S104 of the first embodiment, they differ in that in the former, a corresponding point corresponding to the feature point candidate C_h is selected, rather than a corresponding point corresponding to the first feature point P_i. Since a corresponding point may not be preferably selected for the image T_jg of the object used when adding the feature point candidate C_h, NO_h is less than or equal to NT-1. Following the process in step S306, for each image T_jg of the object recognized from the input photo image I_j, NO_h corresponding points O_hp (p = 1 to NO_h) were found for each feature point candidate C_h. In step S307, the statistics of the physical size of the corresponding points O_hp (p = 1 to NO_h) are calculated for each feature point candidate C_h. In this case, if NO_h corresponding points correspond to a specific feature point candidate C_h, the ratio (NO_h+1) / NT of the occurrence of the relevant feature point in the images of the object to the total number NT of images of the object used to select corresponding points is calculated and determined as a statistic. The statistics are not limited to this example, and the following statistics, for example, can even be calculated. • The averages, variances and probability distributions of the position errors, the errors in the luminance gradient direction and the distances between the feature point candidates C_h and the corresponding points O_hp. To calculate a statistic for the position, the position R_Ohp of the corresponding point O_hp can preferably be converted into a position R_Ohp' when considering the model pattern coordinate system. This can be done using the equation (2) mentioned above. • The average, variance, and probability distribution of the luminance gradient magnitudes of the corresponding points O_hp. • The number of NO_h matching points found. In step S308, based on the calculated statistics, it is determined whether adding each feature point candidate C_h (h = 1 to NC) to the model pattern as a feature point is adequate. For example, if NO_h corresponding points correspond to a specific feature point candidate C_h, the total number of images of the object used in selecting corresponding points is NT, and the ratio of occurrences of the relevant feature point in the images of the object is consequently (NO_h+1) / NT. If the ratio of occurrences of the relevant feature point in the images of the object is lower than a predetermined threshold, it might be said that the feature point candidate does not occur frequently in the images of the object, and adding this feature point candidate to the model pattern as a feature point might therefore be considered inadequate. If other statistics, such as the variance of positions or orientations, are used, and the variance of the positions or orientations of the corresponding points O_hp (p = 1 to NO_h) corresponding to a particular feature point candidate C_h is greater than a predetermined threshold, it may be said that the particular feature point candidate does not represent a feature point that occurs stably in the images of the object, and adding this feature point candidate to the model pattern as a first feature point may consequently be considered inadequate. In step S309, the physical quantity of a feature point candidate C_h, which was determined to be adequate in step S308, is overwritten for each feature point candidate C_h (h = 1 to NC) with the statistics of the physical quantity of the corresponding points O_hp (p = 1 to NO_h) that correspond to the feature point candidate C_h. For example, the position, orientation, luminance gradient magnitude, etc., of the feature point candidate C_h are overwritten with the averages of the position, orientation, luminance gradient magnitude, etc., of the corresponding points O_hp (p = 1 to NO_h). The processing described above adds points with the same position / orientation as different feature point candidates when viewed in the model pattern coordinate system. In step S310, duplicate points are removed from the feature point candidates C_h (h = 1 to NC). As shown in Fig. 9, for example, dimensional space defined by the model pattern coordinate system is subdivided into blocks B for each pixel, and feature point candidates are removed so that only one feature point candidate remains in each block. Although various methods are available as a criterion for removal, for example, a feature point candidate relevant to corresponding points found in large numbers can be made to remain, a feature point candidate with a high average luminance gradient magnitude can be made to remain, or these methods can be used in combination. Considering how the feature point candidate in question is associated with feature point candidates and first feature points in neighboring pixels, even the highest adequate feature point candidate can be made to remain.For example, if the luminance gradient directions of remaining feature point candidates differ significantly between adjacent pixels, one of the feature point candidates may be incorrect. This processing is performed for all blocks containing feature point candidates. In step S311, feature point candidates C_h (h = 1 to NC) that were not removed in the processing mentioned above are added to the model pattern as the first feature points and stored in the model pattern storage unit 26. The second embodiment is not limited to the process mentioned above and can be considered to take on various forms. Although second characteristic points Q_jk that do not correspond to any of the first characteristic points P_i are selected as non-corresponding points O_jgn in the aforementioned step S304, the second characteristic points Q_jk can be compared not only with the first characteristic points P_i, but also with the already added characteristic point candidates C_h, so that second characteristic points Q_jk that correspond to first characteristic points P_i are selected as corresponding points for the characteristic points P_i, those that correspond to characteristic point candidates C_h are selected as corresponding points for the characteristic point candidates C_h, and those that correspond to neither first characteristic points P_i nor characteristic point candidates C_h are selected as non-corresponding points O_jgn.This prevents second feature points Q_jk, which are more likely to be duplicates of second feature points that have already been added as feature point candidates C_h, from being added as feature point candidates C_h. According to the second embodiment, second feature points that are missing in the initially generated model pattern, but occur frequently and consistently in the images of the object, can be automatically added to the model pattern. This produces the following advantageous effects. The initially generated model pattern comprises only feature points extracted from a single image of the object captured in a photograph. However, feature points cannot always be extracted from the entire object if a section is missed during a noticeable luminance gradient due to, for example, variations in light or noise. Consequently, parts missing from the initial model pattern can be compensated for by adding feature points not present in the initial model pattern but found in a large number of images of the object, using the method described in the second embodiment. This can improve robustness and detection accuracy. A third embodiment will be described next. In the third embodiment, determination indices are added to a first feature point forming the model pattern based on the statistics of the physical quantities of corresponding points that correspond to a feature point forming a model pattern and are used in an object recognition unit 23 to improve the recognition reliability. Fig. 10 is a flowchart illustrating processing according to the third embodiment. Steps S401 to S404 are identical to steps S101 to S104 in the first embodiment. Points corresponding to NO_i, determined to correspond to the i-th first feature point P_i forming the model pattern, were found, as at the end of step S104. The m-th corresponding point, which corresponds to the i-th first feature point P_i of the model pattern, is defined as O_im (m = 1 to NO_i). In step S405, the statistics of the physical quantities of the points O_im corresponding to the first feature point P_i are calculated. The following statistics are calculated, for example. This calculation can be performed as in step S307 of the second embodiment. 1. The standard deviation of a distance d_im between the first feature point P_i and the corresponding point O_im is calculated. Assuming that the distance d_im varies (m = 1 to NO_i) according to a normal distribution, a standard deviation σd of the distance d_im can be calculated using a well-known method. 2. The mean and standard deviation of the luminance gradient magnitudes are calculated.Assuming that the magnitude of the luminance gradient maintained at the corresponding point O_im varies according to a normal distribution, a mean µα and a standard deviation σα can be calculated from the luminance gradient at the corresponding point O_im (m = 1 to NO_i). 3. The probability distribution of the luminance gradient magnitude is calculated. Assume that the luminance gradient magnitude varies according to a mixture of normal distributions. A mixture of normal distributions p(x) with the magnitude of the luminance gradient at the corresponding point O_im (m = 1 to NO_i) as a probability variable is given by the following: The parameters πb, µαbund σαb(b = 1 to NB) in the equation mentioned above can be obtained using a well-known procedure. In step S406, determination indices W_i are obtained from the calculated statistics. The following determination indices can each be obtained using the statistics calculated in step S405. (a) The distance tolerance is calculated from the standard deviation σd of the distance d_im. A determination index indicates that a mismatch has occurred if the distance is greater than or equal to 3σd or less than or equal to -σd. Since first feature points with a high σd involve variations in the position of second feature points of the object to be detected, they exhibit a tolerance for large variations, while first feature points with a low σd exhibit a tolerance only for relatively small variations. (b) The mean µα and the standard deviation σα of the luminance gradient magnitude are used to tolerate only second feature points with luminance gradient magnitudes from µα-2σα to µα+2σα.(c) The probability distribution p(x) of the luminance gradient magnitude is used to set a determination index that indicates whether the probability of taking a particular luminance gradient magnitude is greater than or equal to a predetermined value. If the probability of taking a particular luminance gradient magnitude is greater than a given limit p_min, the tolerance is determined. In this case, the limit p_min and the parameters πb, µα, and σαb (b = 1 to NB), which define a mixture of normal distributions, are included in the determination index. In step S407, the obtained determination indices W_i are stored in a model pattern storage unit 26 in association with the first feature point P_i. Instead of limit values ​​obtained from the statistics, the statistics themselves can be stored as determination indices. The processes in steps S404 to S407 are performed for each feature point P_i (i = 1 to NP) that forms the model pattern. The determination indices are stored in the model pattern storage unit 26 in association with the feature points P_i that form the model pattern, in a form such as that shown in Fig. 16. In step S408, the object recognition unit 23 recognizes the object using the determination indices stored in the model pattern storage unit 26. One method for using determination indices in the object recognition unit 23 is to obtain an image T_x of the object, then calculate the degree of similarity with each image of the object and exclude an image T_x of the object with a low degree of similarity. This method is described below. 1. A new image Inew is acquired. 2. An object is recognized from the image Inew. An arbitrary procedure is used for recognition, as in step S103. After this operation, NTX images T_x (x = 1 to NTX) of the object have been obtained. The following operations in 3 to 6 are performed for all images T_x of the object. 3.Second feature points Qnew_k, corresponding to a first feature point P_i that forms the model pattern, are selected from second feature points Qnew_k (k = 1 to NQ) extracted from the images Inew, based on the position R_Tx of the image T_x of the object detected from the image Inew. 4. It is verified whether the selected second feature points are adequate as corresponding points for the first feature point P_i. Determination indices associated with P_i are used. For example, a selected second feature point is determined to be adequate if the luminance gradient magnitude β of this second feature point falls within the range of µα-2σα to µα+2σα (see step S408 for the definition of µ, α, and σ). Adequate corresponding points are stored in the recognition result storage unit 27 as corresponding points O_i for P_i. 5.The processes described in sections 3 and 4 above are performed for all feature points P_i (i = 1 to NP) so that NO corresponding points can be selected. 6. The number of NP of initial feature points forming the model pattern and the number of NO of found corresponding points are used to obtain NO / NP, so that the degree of agreement between the model pattern and the image T_x can be represented using values ​​from 0.0 to 1.0. Alternatively, the individual degrees of agreement between a given first feature point P_i and its corresponding points O_i can be calculated as S_i to determine S_i / NP as the degree of agreement between the model pattern and the image T_x. The individual degrees of agreement S_i can be obtained using the physical quantities of both the first feature point P_i and the corresponding points O_i. For example, if d_i is the distance between the physical quantity of the first feature point P_i and the corresponding point O_i, then S_i, by defining a specific constant d_max, can be obtained as follows: where |d_i| is a value less than or equal to d_max. 7. If the calculated degree of similarity between the model pattern and the image T_x is less than a predetermined limit Smin, the image T_x of the object is excluded from the recognition results. The procedure for using identification indices in the object recognition unit is not limited to this. For example, even the ICP algorithm includes the step of selecting appropriate points for the initial feature points that form a model pattern. In this step, a match / dismatch between the initial feature points and the subsequent feature points can be determined using identification indices. This allows for a more precise determination of the object recognition unit using appropriate points. According to the third embodiment, determination indices that are optimal for object detection can be provided for each feature point forming the model pattern and used to detect an object. This produces the following advantageous effects. Depending on the portion of the first feature point that forms the model pattern, the positional variation can be large or small. If the positional variation is small, a second feature point located at a distance greater than or equal to the variation is more likely to be an inadequate corresponding point. An incorrect match can be prevented by avoiding the use of such second feature points as corresponding points. An incorrect match of corresponding points can negatively impact detection accuracy or lead to a false detection. Furthermore, if the positional variation is large, tolerating even small variations is likely to hinder the detection of corresponding points for the first feature point. This can lower the degree of match, resulting in a failure to detect an object.Preventing such adverse effects can improve the robustness and accuracy of the detection result. The determination indices stored in the model pattern storage unit 26 can also be used in the unit for selecting corresponding points, as in, for example, the first embodiment. Adequacy can be assessed, for instance, using the determination indices in the step of evaluating whether the second feature points Q_jk are adequate as corresponding points for the first feature point P_i, in the unit for selecting corresponding points. 1. According to the third embodiment, determination indices are provided for each feature point P_i (i = 1 to NP) that forms the model pattern. In this case, the determination index used indicates whether the distance di between the first feature point and the second feature point falls within the range of -3σd to 3σd. 2.The unit for selecting appropriate points assesses whether the distance di between the first feature point P_i and the second feature point Q_jk falls within the range of -3σd to 3σd. The second feature point Q_jk is determined to be adequate if the distance di falls within this range; otherwise, it is determined to be inadequate. 3. Subsequent operations are performed using only second feature points Q_jk that have been determined to be adequate as appropriate points. This can calculate the statistics of a predetermined physical quantity using precisely determined corresponding points. Although the first to third embodiments require that the vision sensor uses a camera and that the data input from the vision sensor is a photographic image, the present invention is also applicable when the vision sensor is implemented in a three-dimensional sensor. Three-dimensional sensors include those that measure three-dimensional points on the plane of an object, as in the light sectioning method, and those that measure three-dimensional points on the contour edges of an object, as in the stereo method. Individual three-dimensional points forming a set of three-dimensional points output by the three-dimensional sensor can be directly used as first feature points forming a model pattern according to the present invention. A model pattern can preferably be generated using three-dimensional points located within the marked region as first feature points, as in the first embodiment. A set of three-dimensional points representing a model can also be obtained from CAD data that represents the shape of an object. Three-dimensional points on a plane can be obtained from the CAD data at a predetermined interval, or three-dimensional points on a contour boundary can be obtained at a predetermined interval. If three-dimensional points forming the input data are located on the plane of an object, not only the position of the feature point but also the direction of a perpendicular can be used as physical quantities of each feature point. An exemplary procedure for obtaining perpendiculars in a set of three-dimensional points is well known in, for example, Niloy J. Mitra et al., “Estimating Surface Normals in Noisy Point Cloud Data”, SCG'03 Proceedings of nineteenth annual symposium on Computational geometry, pages 322–328, 2003. If three-dimensional points forming the input data are located on the contour boundary of an object, a direction vector connecting feature points adjacent to the contour boundary can be used as the physical quantity of each feature point. The direction of a perpendicular and a direction vector can be used to describe the position of the feature point. If the three-dimensional sensor uses not only three-dimensional points but also luminance or color information, this information can be added as a physical property of the feature point. A luminance gradient can, of course, even be calculated as a physical property of the feature point using the luminances of neighboring three-dimensional points. Feature points in a set of three-dimensional points can take on other forms, such as SHOT, which is described in Federico Tombari et al., “Unique Signatures of Histograms for Local Surface Description”, Computer Vision, ECCV 2010, Part III., LNCS 6313, pp. 356-369, 2010. Even if the input data represents a set of three-dimensional points, it is possible to recognize an object and obtain its position. Examples include a method as described in Federico Tombari et al., “Object recognition in 3D scenes with occlusions and clutter by Hough voting”, IEEE 2010 Fourth Pacific-Rim Symposium on Image and Video Technology. Even if the input data again represents a set of three-dimensional points, the ICP algorithm can be used. The method for recognizing an object from a set of three-dimensional points is, of course, not limited to such examples. If the input data represents a set of three-dimensional points on a plane, the unit for selecting corresponding points obtains the distance between the position R_Pi of the first feature point P_i and the position R_Qjk' of the second feature point Q_jk when viewed from the model pattern coordinate system using the following procedure: The distance between two points R_Pi and R_Qjk' is simply obtained. A node between a perpendicular to the first feature point P_i and a plane obtained from a set of adjacent points of the second feature point Q_jk is obtained to determine the distance between the node and the first feature point. Although several feature points form a model pattern in the first through third embodiments, it is possible that not several feature points, but rather a single feature point, constitutes a model pattern. If only one feature point forms a model pattern, adjusting the position of the feature point, as in the first embodiment, is meaningless. This is because adjusting the position of the model pattern leads to a change in the relative positional relationship between feature points. However, such an adjustment is only relevant for a single feature point when adding a feature point or changing the determination index. Using only one feature point, for example, can result in too many detection outcomes.In such a case, the detection results can be more reliably limited by adding a matched feature point, as in the second embodiment, or by changing the determination index, as in the third embodiment, in order to optimize the model pattern. However, using only one feature point can pose practical problems, resulting in too many candidates for the detection results. For example, if an edge point is used as a feature point, using only one feature point can result in too many matched detection results. However, using a feature point, such as SIFT, can keep the number of matched feature points small to a certain extent. In the first through third embodiments, after an initial model pattern is generated, an input photograph is selected, and an object is recognized, for example, by the recognition step of a robotic device shown in Fig. 3. This can correct the initially set model pattern based on a photograph obtained through an actual operation. In this case, the range of match for object recognition is set relatively wide until a predetermined number of operations are completed after a model pattern has been set, and an object falling within a relatively wide error range is identified as an object. Each time a predetermined number of operations are performed, the model pattern is corrected to one more suitable for an actual object.After a predetermined number of operations, the range of agreement based on the corrected model pattern can be narrowed or otherwise adjusted to increase the probability of a correct determination. Conversely, the range of agreement can, of course, be expanded according to the actual conditions of an object. If the model pattern is corrected based on a photographic image after the aforementioned actual work step, the robot device can, during corrective processing of the model pattern, reflect the success or failure of a work step performed based on the recognition result obtained from the image processing device. In an image processing device according to a fourth embodiment, which is described next, the success or failure of a work step performed based on the recognition result obtained from the image processing device is reflected during corrective processing of the model pattern. Fig. 19 is a flowchart illustrating processing in the image processing device according to the fourth embodiment. In the image processing device according to the fourth embodiment, an image of an object is only reflected in a correction of the model pattern if a work step based on the recognition result obtained from the image processing device of the robot device is successful, and an image of an object is not reflected in a correction of the model pattern if such a work step has failed. The generation of a model pattern in step S501 is the same as in other embodiments. The image input in step S502 and the object recognition in step S503 are also the same as in other embodiments and are performed for each operation step that uses the recognition result obtained from the image processing device. In step S504, a work step is performed based on the object recognition result from step S503. As a result of this work step, its success or failure is determined; for example, a failure is assumed to be due to a problem with the object recognition result. In step S505, the acceptability of the recognition result is determined, and if this result is acceptable (if the operation step is successful), the process continues with step S506; otherwise (if the operation step has failed), the process returns to step S507. In step S506, a correction processing of the model pattern is performed in steps S104 to S106 of the first embodiment, steps S304 to S311 of the second embodiment or steps S404 to S407 of the third embodiment and the process returns to step S502. In step S507, a predetermined handling operation is performed for a recognition failure (operation step failure), such as removing an object, which results in an operation step failure or recording the photo image, and the process then returns to step S502. According to the present invention, an object can be precisely detected even if the physical sizes of feature points of the object vary.

Claims

Image processing device (21) that recognizes an image of an object from input data acquired by a vision sensor (10) based on a model pattern comprising a set of several first feature points representing a shape of the object, the device comprising: an object recognition unit (23) that, for each of several input data values ​​obtained by acquiring the object, recognizes an image of the object by matching several second feature points extracted from the input data with several first feature points forming the model pattern; a unit (24) for selecting corresponding second feature points corresponding to the several first feature points forming the model pattern for each of the images of the object recognized from the several input data values, from the several second feature points forming the image.selects and stores the selected second feature points as corresponding points in association with the first feature points; and a model pattern correction unit (25) which, for each of the multiple first feature points forming the model pattern, calculates a statistic of a predetermined physical quantity of the multiple corresponding points associated with the first feature point and corrects the first feature point on the basis of the calculated statistic of the predetermined physical quantity, wherein the unit (24) for selecting corresponding points for each of the at least one recognized image of the object, stores at least one second feature point that does not correspond to any of the first feature points forming the model pattern, stores multiple second feature points forming the image as a feature point candidate, and stores a second feature point corresponding to the at least one stored feature candidate.selects and stores the selected second feature point as a corresponding point in association with the feature point candidate, and the model pattern correction unit (25) further computes for each of the at least one stored feature point candidate a statistic of a predetermined physical quantity of several corresponding points associated with the feature point candidate and adds the feature point candidate to the model pattern as a first feature point based on the statistic of the predetermined physical quantity. Image processing device (21) according to claim 1, wherein the vision sensor (10) comprises a camera and the input data includes a grayscale image and a color image. Image processing device (21) according to claim 1, wherein the vision sensor (10) comprises a three-dimensional sensor and includes the input data of an area image and a set of three-dimensional points that serves as a point cloud. Image processing device (21) according to claim 1, wherein the model pattern correction unit (25) excludes an inadequate corresponding point from the multiple corresponding points associated with the first feature point and calculates a statistic of the predetermined physical quantity using only one adequate corresponding point.

Images