Control method, control program, recording medium, article manufacturing method, and system

Abstract

To solve the problem in which a technique has been required which can stably obtain information necessary to properly execute a visual servo from a captured image.SOLUTION: A control method for controlling a device including a movable part by a visual servo includes: acquiring a target image according to a target relative position relation between a movable part and an object; extracting a target feature point whose freedom degree of information is ψ from the target image by image processing; acquiring a current image according to a current relative position relation between the movable part and the object; extracting a candidate feature point from the current image by image processing; extracting a current feature point associated with the target feature point from candidate feature points by performing matching processing between the candidate feature points and the target feature points; and generating a control signal for moving the movable part at a freedom degree n using matching information. The number N of the target feature points extracted from the target image satisfies N≥2×L when the minimum integer M satisfying N>(n+1) / ψ is L.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a control method for controlling the operation of a device having a moving part by visual servoing, and the like. [Background technology]

[0002] Robots are increasingly being introduced in various fields, including production sites. Visual servoing, which measures changes in the position of a target object as visual information and uses that information as feedback, is one method for controlling the position and posture of a robot. Specifically, a target image of the robot and / or work object is captured at a target position and posture, and a current image is captured at the current position and posture, and the robot is controlled based on the difference between these images. This method has the advantage of eliminating the need for strict calibration in advance, and allowing positioning to be performed based on the captured images even if there is an error in the robot's actual movement relative to the command values.

[0003] Visual servoing can be broadly divided into two types depending on the control method. One is position-based visual servoing, which recognizes the current position of the object and feeds back to the robot the difference between the target position and the current position. The other is image-based visual servoing, which extracts image features contained in the image of the object in the current image and feeds back the difference between the image features in the target image. Of these, image-based visual servoing uses the image Jacobian to associate image feature deviations with robot control amounts. This method has the advantage of being computationally less demanding because it does not require recognition of the object's position and orientation, and moreover, since recognition errors are less likely to be included in the feedback signal, it allows for highly accurate control of the robot.

[0004] Patent Document 1 discloses a technique for capturing an image of an object equipped with markers, extracting the markers in the image as image features, and performing visual servo control. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-254518 Summary of the Invention [Problem to be solved by the invention]

[0006] To perform appropriate control using image-based visual servoing, it is necessary to be able to properly extract the visual features of the target object from the current image captured at the current position and orientation. However, it may be difficult to extract the visual features of the target object from the current image captured under real-world conditions. For example, it may be difficult to extract the visual features of the target object if it is not permissible to attach a marker or other label that is highly detectable in an image, or if the target object does not have any prominent external features. If some or all of the visual features of the target object cannot be extracted, it may be impossible to generate an appropriate feedback signal to control the robot, which may result in erroneous control or the need to stop visual servoing.

[0007] Therefore, there was a need for technology that could stably obtain the information necessary to properly implement visual servoing from captured images. [Means for solving the problem]

[0008] A first aspect of the present invention is a control method for controlling a device having a movable part by visual servoing, the method comprising: acquiring a target image according to a relative positional relationship between the movable part and a target of an object; and performing image processing to obtain a target image from the target image. Rame extracting target feature points, acquiring a current image corresponding to the current relative positional relationship between the movable part and the object, extracting candidate feature points from the current image by image processing, performing a matching process between the candidate feature points and the target feature points, extracting current feature points corresponding to the target feature points from the candidate feature points, and using matching information based on the matching between the current feature points and the target feature points by the matching process ,beforegenerating a control signal for moving the movable part; a first clustering process classifying the target feature points into a plurality of target feature point clusters and obtaining target feature point cluster representative information for each of the plurality of target feature point clusters; a second clustering process classifying the current feature points into a plurality of current feature point clusters and obtaining current feature point cluster representative information for each of the plurality of current feature point clusters; and the matching information includes the current feature point cluster representative information and the target feature point cluster representative information. The control method is characterized by the above.

[0009] A second aspect of the present invention is a control method for controlling a device having a movable part by visual servoing, the method comprising: acquiring a target image according to a relative positional relationship between the movable part and a target of an object; and performing image processing to obtain a target image from the target image. Rame extracting target feature points, classifying the target feature points into a plurality of target feature point clusters by a first clustering process, acquiring target feature point cluster representative information for each of the plurality of target feature point clusters, acquiring a current image according to a current relative positional relationship between the movable part and the object, extracting candidate feature points from the current image by image processing, performing a matching process between the candidate feature points and the target feature points to extract current feature points associated with the target feature points from the candidate feature points, classifying the current feature points into a plurality of current feature point clusters by a second clustering process, acquiring current feature point cluster representative information for each of the plurality of current feature point clusters, and using matching information between the current feature point cluster representative information and the target feature point cluster representative information ,before A control signal for moving the moving part is generated. Do, this The control method is characterized by the above.

[0010] A third aspect of the present invention is a control method for controlling a device having a movable part by visual servoing, the method comprising: acquiring a target image according to a relative positional relationship between the movable part and a target of an object; and performing image processing to obtain a target image from the target image. Rame extracting target feature points, acquiring a current image corresponding to the current relative positional relationship between the movable part and the object, extracting candidate feature points from the current image by image processing, performing a matching process between the candidate feature points and the target feature points, extracting current feature points corresponding to the target feature points from the candidate feature points, correcting the current feature points using priorities set for the current feature points and the target feature points obtained by the matching process, and using matching information between the corrected current feature points and the target feature points ,before A control signal for moving the moving part is generated. Do, thisThe control method is characterized by the above.

[0011] A fourth aspect of the present invention is a system including a device having a movable part and a control part, wherein the control part acquires a target image according to a relative positional relationship between the movable part and a target of an object, and performs image processing to obtain a target image from the target image. Rame extracting target feature points, acquiring a current image corresponding to the current relative positional relationship between the movable part and the object, extracting candidate feature points from the current image by image processing, performing a matching process between the candidate feature points and the target feature points, extracting current feature points corresponding to the target feature points from the candidate feature points, and using matching information based on the matching between the current feature points and the target feature points by the matching process ,before generating a control signal for moving the movable part; a first clustering process classifying the target feature points into a plurality of target feature point clusters and obtaining target feature point cluster representative information for each of the plurality of target feature point clusters; a second clustering process classifying the current feature points into a plurality of current feature point clusters and obtaining current feature point cluster representative information for each of the plurality of current feature point clusters; and the matching information includes the current feature point cluster representative information and the target feature point cluster representative information. The system is characterized by the above.

[0012] A fifth aspect of the present invention is a system including a device having a movable part and a control part, wherein the control part acquires a target image according to a relative positional relationship between the movable part and a target of an object, and performs image processing to obtain a target image from the target image. Rame extracting target feature points, classifying the target feature points into a plurality of target feature point clusters by a first clustering process, acquiring target feature point cluster representative information for each of the plurality of target feature point clusters, acquiring a current image according to a current relative positional relationship between the movable part and the object, extracting candidate feature points from the current image by image processing, performing a matching process between the candidate feature points and the target feature points to extract current feature points associated with the target feature points from the candidate feature points, classifying the current feature points into a plurality of current feature point clusters by a second clustering process, acquiring current feature point cluster representative information for each of the plurality of current feature point clusters, and using matching information between the current feature point cluster representative information and the target feature point cluster representative information ,before A control signal for moving the moving part is generated. Do, this This is a system characterized by the following.

[0013] A sixth aspect of the present invention is a system including a device having a movable part and a control part, wherein the control part acquires a target image according to a relative positional relationship between the movable part and a target of an object, and performs image processing to obtain a target image from the target image. Rame extracting target feature points, acquiring a current image corresponding to the current relative positional relationship between the movable part and the object, extracting candidate feature points from the current image by image processing, performing a matching process between the candidate feature points and the target feature points, extracting current feature points corresponding to the target feature points from the candidate feature points, correcting the current feature points using priorities set for the current feature points and the target feature points obtained by the matching process, and using matching information between the corrected current feature points and the target feature points ,before A control signal for moving the moving part is generated. Do, this This is a system characterized by the following. [Effects of the Invention]

[0014] According to the present invention, information required for properly performing visual servoing can be stably acquired from a captured image. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of a robot system according to a first embodiment. [Figure 2] 10 is a flowchart showing the flow of processing for setting target feature points in the first embodiment. [Figure 3] 5 is a flowchart showing the flow of processing for controlling the movement of the robot 1 by visual servo control using information on target feature points in the first embodiment. [Figure 4] 10(a) is a diagram showing an example of a user interface screen that can be used in embodiment 1. FIG. 10(b) is an example of a screen that displays a matching result on the output device 103 in embodiment 1. FIG. [Figure 5](a) An example of a target image acquired in embodiment 1. (b) An example showing a feature point FP1 extracted from the target image superimposed on the target image in embodiment 1. (c) An example showing a target feature point F100 extracted in embodiment 1 superimposed on the target image. [Figure 6] (a) An example of a current image acquired in embodiment 1. (b) An example showing candidate feature point FP2 extracted in embodiment 1 superimposed on the current image. (c) An example showing current feature points extracted from the candidate feature points in embodiment 1 superimposed on the current image. (d) A diagram showing the results of matching processing between target feature point F100 and candidate feature point FP2 in embodiment 1. [Figure 7] (a) An example of a target image for conventional visual servo control. (b) An example of a target feature point used in conventional visual servo control. (c) An example of a current feature point extracted by conventional visual servo control. (d) A diagram for explaining the correspondence between a current feature point F101 and a target feature point in conventional visual servo control. [Figure 8] FIG. 10 is a schematic diagram showing the general configuration of a robot system according to a second embodiment. [Figure 9] 10 is a flowchart showing the flow of processing for setting target feature points in the second embodiment. [Figure 10] 10 is a flowchart showing the flow of processing for controlling the movement of the robot 1 by visual servo control using information on target feature points in the second embodiment. [Figure 11] 10A is a diagram showing an example of a user interface screen that can be displayed in each processing step in embodiment 2. FIG. 10B is a diagram showing an example of a display screen when a matching result is displayed on an output device 103. FIG. [Figure 12](a) An example of a target image acquired in embodiment 2. (b) An example showing feature points FP1 extracted from the target image superimposed on the target image in embodiment 2. (c) An example showing target feature points F100 extracted in embodiment 2 superimposed on the target image. (d) An example showing the results of clustering target feature points superimposed on the target image in embodiment 2. (e) An example showing target feature point cluster representative information calculated in embodiment 2 superimposed on the target image. [Figure 13] (a) An example of a current image acquired in embodiment 2. (b) An example showing candidate feature points FP2 extracted in embodiment 2 superimposed on the current image. (c) An example showing a schematic diagram of the matching process between target feature points and candidate feature points in embodiment 2. (d) An example showing current feature points extracted from among candidate feature points in embodiment 2 superimposed on the current image. (e) An example showing the results of clustering current feature points in embodiment 2 superimposed on the current image. (f) An example showing current feature point cluster representative information calculated in embodiment 2 superimposed on the current image. [Figure 14] FIG. 10 is a schematic diagram showing a matching process between representative information of a target feature point cluster and representative information of a current feature point cluster in the second embodiment. [Figure 15] FIG. 10 is a schematic diagram showing a schematic configuration of a robot system according to a third embodiment. [Figure 16] 11 is a flowchart showing the flow of processing for setting target feature points in the third embodiment. [Figure 17] 11 is a flowchart showing the flow of processing for controlling the movement of the robot 1 by visual servo control using information on target feature points in the third embodiment. [Figure 18] 10A is a diagram showing an example of a user interface screen that can be displayed in each processing step in embodiment 3. FIG. 10B is a diagram showing an example of a display screen when a matching result is displayed on the output device 103. FIG. [Figure 19](a) An example of a target image acquired in embodiment 3. (b) An example in which feature point FP1 extracted from the target image is superimposed on the target image in embodiment 3. (c) An example in which target feature point F100 extracted in embodiment 3 is superimposed on the target image. (d) An example in which target feature point F100 and priority order are superimposed on the target image in embodiment 3. [Figure 20] (a) An example of a current image acquired in embodiment 3. (b) An example showing candidate feature point FP2 extracted in embodiment 3 superimposed on the current image. (c) An example showing current feature points extracted from the candidate feature points in embodiment 3 superimposed on the current image. (d) An example showing the results of performing interpolation processing on current feature points, and deleting and adding them in embodiment 3. DETAILED DESCRIPTION OF THE INVENTION

[0016] The following describes a control method, a system, and the like according to embodiments of the present invention with reference to the drawings. The embodiments described below are merely examples, and those skilled in the art can appropriately modify and implement the detailed configurations within the scope of the present invention.

[0017] In the drawings referred to in the following description of the embodiments, elements denoted by the same reference numerals have the same functions unless otherwise specified. Furthermore, the drawings may be represented schematically for the convenience of illustration and explanation, and therefore may not strictly correspond to the actual shapes, sizes, arrangements, etc.

[0018] [Embodiment 1] (system) 1 is a schematic diagram showing a schematic configuration of a robot system according to embodiment 1. The robot system includes a robot 1 as a device having a movable part, and a controller 100 as a control unit.

[0019] The robot 1 is, for example, a multi-axis controlled articulated robot, but the robot 1 may be any other type of robot. The device having a movable part may also be a device other than a robot, such as a device with a movable part that can perform movements such as extension, contraction, bending, vertical movement, horizontal movement, or rotation, or a combination of these movements. A movable part refers to a part that can move relative to a reference point, where movement means displacement, and displacement means a change in position. Therefore, a movable part is a part whose position can be changed relative to a reference point. Here, the reference point may be another part of the device having the movable part, or it may be something separate from the device having the movable part, or it may be the Earth. The entire device having the movable part may be movable relative to a reference point, and the device having the movable part may be a moving object, such as a vehicle such as an automobile, a ship, or an aircraft such as a drone.

[0020] The illustrated robot 1 includes an end effector 2 and an imaging device 3, and is connected to a controller 100 via wired or wireless communication so that they can communicate with each other. The robot 1 can perform tasks such as inspecting a workpiece 4 using the imaging device 3, or moving or positioning the workpiece 4 using the end effector 2. Alternatively, the robot 1 can perform a variety of tasks, including tasks related to the manufacture of goods, such as grasping the workpiece 4 and assembling it with other workpieces, or using the end effector 2 to operate a tool to process the workpiece 4.

[0021] The controller 100 as a control unit is a device that controls the operation of the robot 1, and is composed of hardware such as a control device 101, an input device 102, an output device 103, and a storage device 104.

[0022] The control device 101 can send and receive signals to and from the motors, sensors, encoders, etc. in the robot 1 and the imaging device 3 via a built-in I / O port. From the hardware perspective, the control device 101 is a computer configured with a CPU (Central Processing Unit), various storage devices such as RAM, ROM, HDD, SSD, etc., I / O ports, dedicated circuits such as ASIC, etc. The control device 101 may have a hardware configuration suitable for high-speed processing by including a GPU (Graphics Processing Unit) and GPU memory (VRAM). It is also possible to configure the control device 101 by connecting multiple computers through a network.

[0023] FIG. 1 does not show the hardware configuration of the control device 101, but rather shows the functions of the control device 101 as functional blocks. While FIG. 1 uses functional blocks to represent functional elements necessary for explaining the features of this embodiment, general functional elements that are not directly related to the problem-solving principle of the present invention are omitted. Furthermore, the functional elements shown in FIG. 1 are conceptual functional elements and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of the functional blocks is not limited to the illustrated example, and all or part of them can be functionally or physically distributed and integrated in any unit depending on the usage situation, etc.

[0024] The control device 101 includes the following functional blocks: an image acquisition unit 105, a feature point extraction unit 106, a feature matching unit 107, and a robot control unit 108. These functional blocks are realized by a CPU or GPU reading and executing a program stored in a computer-readable recording medium. The computer-readable recording medium may be a ROM, a storage device 104, or an external storage device (not shown). For example, a hard disk, a flexible disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory such as a USB memory, or an SSD may be used. Note that some of the functional blocks may be realized by dedicated hardware such as an ASIC.

[0025] The functions of each unit will be described in detail later, but the image acquisition unit 105 has a function of having the imaging device 3 capture and acquire images such as a target image and a current image. The feature point extraction unit 106 has a function of extracting feature points from the captured image. The feature matching unit 107 has a function of performing a matching process between the current feature points and the target feature points to generate matching information. The robot control unit 108 has a function of generating a control signal for the robot 1 from the matching information generated by the feature matching unit 107 and controlling the operation of the robot 1.

[0026] The input device 102 included in the controller 100 is a device operated by a user to input instructions and information to the control device 101. For example, a mouse, keyboard, touch panel, jog dial, pointing device, dedicated console, or voice input device may be used as a user interface. The input device 102 may also include a device for inputting information to the control device 101 from an external device (not shown), and may use a network adapter for receiving information from an external network.

[0027] The output device 103 is a display device that displays information acquired by the control device 101 and a UI screen for the user to input information to the control device 101. For example, a liquid crystal display or an organic EL display may be used as a user interface. The output device 103 may also include a device for outputting information to an external device (not shown), and may use a network adapter or the like for transmitting information to a computer or storage device on an external network.

[0028] The storage device 104 is a device that stores and reads various setting information (imaging device settings, feature point extraction parameters, robot control programs, etc.) that the controller 100 refers to when performing visual servo control, target images, robot control logs, etc. For example, it can be configured with devices such as HDD, SSD, flash memory, and network storage.

[0029] Note that FIG. 1 merely shows one example of the configuration of a robot system. In this example, the imaging device 3 is fixed to a link of the robot 1, but the imaging device 3 may also be installed external to the robot 1. The imaging device 3 may move in conjunction with the movement of the movable part of the robot 1, or may move independently of the movement of the movable part of the robot 1. If the imaging device 3 moves in conjunction with the movement of the movable part of the robot 1, the relative positional relationship between the imaging device 3 and the movable part can be determined by knowing the relative positional relationship between the imaging device 3 and the movable part, and the relative positional relationship between the object photographed by the imaging device 3 and the movable part can be derived from the relative positional relationship between the imaging device 3 and the object. When the imaging device 3 is installed external to the robot 1, the imaging device 3 is installed in a position and orientation that allows it to capture images of the robot 1 and workpiece 4, that is, so that the robot 1 and workpiece 4 are included in the angle of view.

[0030] (Visual Servo Control) The visual servo control according to this embodiment will now be described. In the following example, an end effector 2 and an imaging device 3 are installed on the most distal link of a robot 1, which is a six-axis jointed robot arm, and the robot 1 performs a predetermined operation using visual servo control. For example, the robot 1 uses the end effector 2 to grasp a workpiece 4 and transport it to a predetermined position (for example, a tray not shown). The robot 1 performs a series of operations, including approaching the workpiece 4 to grasp it, grasping it, and transporting the grasped workpiece 4 to the tray.

[0031] All of the motions of the robot 1 from the start to the completion of this series of tasks may be controlled using visual servoing, or only some of the motions within the series of tasks may be controlled using visual servoing. For example, in the latter case, visual servoing may be used for motions that require highly accurate control of the relative positional relationship between the robot 1 and the workpiece 4, while position control based on teaching may be used for motions that allow for relatively low-precision control. Visual servoing may be suitably applied to motions that require highly accurate control of the relative positional relationship between the robot and the workpiece, such as gripping a specific portion of the workpiece with an end effector or placing the gripped workpiece in a predetermined position on a tray. On the other hand, for motions such as moving the robot 1 from the home position to near the workpiece 4 as a preliminary step to the gripping motion, it may be possible to prioritize motion speed using normal position control based on teaching.

[0032] For ease of explanation, for an operation performed under visual servo control, the appropriately set positions and postures among the series of positions and postures that the robot 1 passes through from the start to the completion of the operation will be referred to as target position 1, target position 2, etc. The robot 1 will pass through the positions and postures of target position 1, target position 2, etc. in sequence while performing the operation.

[0033] The position and orientation of the robot 1 at each target position are set so that the imaging device 3 fixed to the link of the robot 1 can capture an image of an object whose relative positional relationship with the robot 1 is to be evaluated in visual servoing. For example, the target position of the robot 1 set for the operation of approaching the workpiece 4 to grasp it is set to a position and orientation that allows the imaging device 3 to capture an image of the workpiece 4 as the target. Also, for example, the target position of the robot 1 set for the operation of transporting the grasped workpiece 4 to a tray is set to a position and orientation that allows the imaging device 3 to capture an image of the tray as the target. This makes it possible to capture a target image corresponding to the relative positional relationship between the robot as a movable part and the workpiece (or tray) as the target. The target image may include information regarding the relative positional relationship between the robot as a movable part and the workpiece (or tray) as the target.

[0034] In addition, when the imaging device 3 is not attached to the robot 1 but is installed at a position separate from the robot 1, it is installed so that it can capture a target image according to the relative positional relationship between the robot as a movable part and the workpiece (or tray) as an object. In other words, the positional relationship between the target position of the robot 1 and the imaging device 3 is set so that at least the workpiece (or tray) and a part of the robot 1 (for example, the end effector 2) are within the angle of view of the imaging device 3 when the robot 1 is at the target position.

[0035] Fig. 2 is a flowchart showing the flow of processing in preparation for visual servo control, in which robot 1 is moved to a target position in advance, an image is captured by imaging device 3, and target feature points are set based on the acquired target image. Fig. 3 is a flowchart showing the flow of processing in which information on the set target feature points is used to control the operation of robot 1 by visual servo control. Each step will be explained below.

[0036] (Target feature point setting process) In preparation for visual servo control, in step S10 of FIG. 2, the robot 1 is moved to an initial target position (e.g., target position 1). For example, a teaching pendant can be connected to the control device 101 as the input device 102, and the user can move the robot 1 to the initial target position while operating the teaching pendant. Alternatively, the user can manually operate the robot 1 to move it to the initial target position by so-called direct teaching. The target position needs to be set to a position and orientation that allows the imaging device 3 fixed to the link of the robot 1 to capture an image of a target object (e.g., workpiece 4). Therefore, in step S10, it is desirable to display an image captured by the imaging device 3 on the display screen of the output device 103 so that the user can operate the robot while checking the captured image.

[0037] Next, in step S11, imaging is performed while the robot 1 is at the target position, and an image is acquired. The image captured at the target position is referred to as a "target image." Specifically, the image acquisition unit 105 drives the imaging device 3 to capture an image and acquire the target image. The target image captured by the imaging device 3 is linked to information about the target position (target position 1, target position 2, ...) where the image was captured, and is treated as, for example, target image 1, target image 2, ...

[0038] The control device 101 can display the captured target image on the display screen of the output device 103 so that the user can confirm it. FIG. 4(a) is an example of a user interface screen, and the display screen may display an image data display section V100, a capture button V101, and a registration button V102. The user can input instructions using the input device 102 to the capture button V101 and the registration button V102, which serve as instruction input sections. FIG. 5(a) shows an example of an acquired target image. For example, if the captured image is overexposed due to sudden external light irradiation, and the user determines that there is a problem after viewing the displayed image, the user can operate the capture button V101 to perform recapture. The target image can be linked to information about the target position and registered in the storage device 104. To register information such as the target image in the storage unit, the user may operate the registration button V102 using the input device 102, or a control program may be configured so that the controller 100 automatically registers the information in the storage device 104.

[0039] Next, in step S12, image processing is performed on the target image acquired in step S11 to extract feature points, which are coordinate information on the image. Fig. 5(b) shows the feature points FP1 extracted from the target image, represented schematically by dots and superimposed on the target image. The number of extracted feature points FP1 may vary depending on the content of the target image and the image processing algorithm, but preferably 200 to 300 feature points FP1 are extracted.

[0040] The image shown in Fig. 5(b) can be displayed in the image data display section V100 of the user interface screen shown in Fig. 4(a). If the user determines from the display screen that there is a problem with the processing results, they can select the capture button V101 to execute steps S11 and S12 again.

[0041] The image features evaluated when extracting feature points can be local features that are invariant to rotation and scaling, such as SIFT, SURF, and AKAZE. For example, when using SIFT as the image feature, key points in the image are extracted using DoG processing, and key points with principal curvatures and contrasts above a preset threshold are extracted as feature points. In particular, when using SIFT, the feature values ​​of each key point can be calculated in preparation for later processing. Alternatively, when using an image template as the image feature, all areas of a specific size extracted from the current image become feature points.

[0042] Next, in step S13, feature points to be referenced during visual servo control (hereinafter referred to as "target feature points") are selected from the feature points extracted in step S12. Points having image features with high discrimination among the extracted feature points, such as points with high corner strength or edge strength, are selected as target feature points. The target feature points may be selected, for example, by the user specifying highly discriminable feature points using the input device 102 while viewing the display screen shown in FIG. 5(b). Alternatively, the feature point extraction unit 106 may evaluate the image features (discriminability) of each feature point, and automatically select only the feature points with the highest evaluation values ​​within a range specified in advance by the user.

[0043] In this embodiment, multiple target feature points are selected so that the number of target feature points to be extracted exceeds the minimum number of feature points L. The minimum number of feature points L is the number of feature points required so that the total sum of the degrees of freedom of information possessed by each feature point is equal to or greater than the number of degrees of freedom required for visual servo control.

[0044] Note that when the number of coordinate axes (degrees of freedom) for feedback control is n (sometimes referred to as n degrees of freedom), and the number of degrees of freedom of information possessed by a target feature point is ψ (sometimes referred to as ψ degrees of freedom), the smallest integer M that satisfies M>(n+1) / ψ can also be expressed as the minimum number of feature points L.

[0045] Here, the degrees of freedom of information possessed by multiple feature points are taken into consideration as follows. Generally, coordinate information on an image is used in visual servo control algorithms. For example, image features (local features) that are invariant to rotation and scaling, such as SIFT, SURF, and AKAZE, can be used. When such image features are used, each feature point has information with two degrees of freedom, i.e., image coordinates (u, v). Therefore, the total degrees of freedom of information possessed by multiple feature points is twice the number of feature points.

[0046] The degrees of freedom required for visual servoing control are the number of coordinate axes (degrees of freedom) for feedback control in the task space, i.e., the coordinate space representing the space in which the robot performs its work, plus one. For example, if the workpiece 4 is stationary and not constrained by guides or other constraints, the coordinate axes for feedback control would naturally be the translational axes (x, y, z) and the rotational axes (ωx, ωy, ωz), and the total number of coordinate axes (degrees of freedom) would be six. In this case, the degrees of freedom required for visual servoing control is the integer obtained by adding one to six, or seven. Note that if the workpiece 4 is constrained by guides or other constraints and its posture is fixed, the coordinate axes for feedback control can be three degrees of freedom (translational axes only) or one degree of freedom (linear movement only), depending on the constraint state. In such cases, the number of degrees of freedom required for visual servoing control is calculated by adding one to the number of coordinate axes for feedback control.

[0047] For example, when performing feedback control with six degrees of freedom, the number of degrees of freedom required for visual servo control is seven, and the total number of degrees of freedom of the information possessed by multiple feature points must be seven or more. If rotation- and scale-invariant local features (two degrees of freedom) are used, the minimum number of feature points L for making the total number of degrees of freedom of the information possessed by the feature points seven or more is four (2 × 4 ≥ 7). In this case, the feature point extraction unit 106 extracts a number exceeding the minimum number of feature points L of four, i.e., five or more target feature points. Figure 5(c) shows an example in which seven target feature points F100 are extracted from a large number of feature points FP1 (Figure 5(b)).

[0048] It is also possible to extract multiple target feature points using image features that are not rotation- or scale-invariant, such as image corners or image templates. When extracting such image features including pose information, each feature point has three or more degrees of freedom. As in this case, the minimum number of feature points L changes depending on how the image features are selected, and the number of target feature points to be extracted may also change accordingly.

[0049] The control program may be configured to re-execute steps S11 and after if the feature point extraction unit 106 determines that the number of target feature points extracted in step S13 does not satisfy a predetermined condition. The predetermined condition is, for example, L+1≦N≦L+40, where N is the number of target feature points.

[0050] For the extracted target feature point, information specifying the image feature corresponding to a specific part of the target object can be linked to coordinate information and distance information on the image. Note that the distance information is the distance from the image sensor surface to the feature point when viewed along the optical axis direction of the image capture device 3, i.e., the z coordinate in the camera coordinate system. If the image capture device 3 is a stereo camera, the distance can be calculated from the stereo images. If not, the distance information can be obtained by measuring using a 3D sensor or by the user actually measuring.

[0051] Next, in step S14, information related to the target feature points selected in step S13 is stored in a storage unit such as the storage device 104 or the main storage device (RAM) of the control device 101 so that it can be referenced in later processing. The control program may be configured so that the feature point extraction unit 106 automatically executes this processing. Alternatively, the control program may be configured so that a user interface screen such as the one shown in FIG. 4(a) is displayed on the display screen of the output device 103, and the user operates the registration button V102 using the input device 102 to receive the command.

[0052] By executing steps S11 to S14, the process of setting a target feature point for one target position is completed, but in step S15, it is determined whether the process of setting target feature points for all target positions is completed. If the process of all target positions is not completed (step S15: NO), the process proceeds to step S16, where the robot 1 is moved to the next target position, and the process from step S11 onwards is performed. If the process of all target positions is completed (step S15: YES), the process of setting target feature points ends.

[0053] Once the setting of the target feature points is complete, the robot 1 is ready to operate under visual servo control, and the robot system may have the robot 1 start actual work, or may have the robot 1 wait until actual work begins at any time. Furthermore, when the robot 1 is made to repeatedly perform the same work operation, such as in the repeated production of the same type of product, the process of setting the target feature points is performed only once, and visual servo control is performed by referring to the target feature points each time the robot performs a work operation.

[0054] (Visual Servo Control) A process for making the robot 1 perform a task using visual servo control will be described with reference to Fig. 3. When visual servo control is started, in step S20, the feature matching unit 107 reads information about the target feature points set at the initial target position from the storage device 104 or the main storage device (RAM) of the control device 101.

[0055] Next, in step S21, the image acquisition unit 105 controls the imaging device 3 to acquire a current image from the imaging device 3 and store it in the main memory device (RAM) of the control device 101. In the following description, the image captured by the imaging device 3 when the robot 1 is in the current position and posture will be referred to as the "current image."

[0056] The angle of view of the imaging device 3 when capturing the current image is set so as to capture an image of an object whose relative positional relationship with the robot 1 is to be evaluated. For example, if the work operation is to approach the workpiece 4 in order to grasp it, at least a part of the workpiece 4 is captured as the object in the current image. Also, for example, if the work operation is to transport the grasped workpiece 4 to a tray, at least a part of the tray is captured as the object in the current image. In this way, a current image is captured according to the relative positional relationship between the robot as a movable part and the workpiece (or tray) as the object. Note that if the imaging device 3 is not attached to the robot 1 but is installed at a position separate from the robot 1, a current image is captured by the imaging device 3 according to the relative positional relationship between the robot as a movable part and the workpiece (or tray) as the object.

[0057] The current image may include information regarding the relative positional relationship between the robot as a moving part and the workpiece (or tray) as an object. The target image and the current image may be in a relationship that allows feedback control (visual servoing) of the difference between the target image and the current image. Note that the object captured as the target image and the object captured as the current image do not necessarily have to be the same. For example, the object captured as the target image may be a sample workpiece 4, and the object captured as the current image may be a mass-produced workpiece 4. Furthermore, the device whose relative positional relationship with the object is defined in the target image and the device whose relative positional relationship with the object is defined in the target image do not necessarily have to be the same. For example, the device whose relative position with respect to the object is defined in the target image may be a test robot, and the device whose relative position with respect to the object is defined in the current image may be a mass-production robot. The target image does not have to be a real-life image; it may be a CG (Computer Graphics) image corresponding to the relative position of the robot 1 and the workpiece 4 on a simulator.

[0058] 6(a) shows an example of an acquired current image. In this example, a workpiece 4 is photographed by the imaging device 3. The current image may be displayed on the screen of the output device 103 so that the user can check it, or may be stored in the storage device 104 so that it can be used as data later.

[0059] When the current image is acquired, in step S22, the feature point extraction unit 106 processes the current image to extract multiple feature points. For ease of explanation, the feature points extracted from the current image in step S22 are referred to as "candidate feature points." The method for extracting candidate feature points from the current image can be the same as the method for extracting feature points from the target image in step S12.

[0060] For example, when SIFT is used as the image feature, key points in the image are extracted by DoG processing, and key points with principal curvatures and contrasts equal to or greater than a preset threshold are extracted as candidate feature points. In particular, when SIFT is used, the feature values ​​of each key point may be calculated in preparation for subsequent processing. Alternatively, when an image template is used as the image feature, all areas of a specific size extracted from the current image become candidate feature points. Note that, when an image template is used, if it is easy to simultaneously perform the extraction of candidate feature points and the matching process, steps S22 and S23 may be performed together.

[0061] Fig. 6(b) is an example showing a schematic representation of candidate feature points FP2 extracted from the current image shown in Fig. 6(a) superimposed on the current image. The candidate feature points can be displayed on the screen of the output device 103 for the user to check, or can be stored in the storage device 104 for later use as data.

[0062] Note that, to deal with cases where the current image was not captured under appropriate shooting conditions due to the influence of sudden external light or the like, it may be determined whether the number of candidate feature points FP2 extracted in step S22 satisfies a predetermined condition. If it is determined that the predetermined condition is not satisfied, the control program may be configured to execute step S21 and subsequent steps again. This is because a sufficient number of candidate feature points FP2 must be extracted to ensure the degree of freedom (number of control axes) required for visual servo control. The predetermined condition may be, for example, (minimum number of feature points L+1)≦(number of candidate feature points) or (number of target feature points)<(number of candidate feature points).

[0063] Next, in step S23, a matching process is performed between the candidate feature points and the target feature points to generate matching information. In the following explanation, for convenience, the candidate feature points that can be associated with the target feature points through matching will be referred to as "current feature points." Fig. 6(c) is an example that schematically shows current feature points extracted from the candidate feature points shown in Fig. 6(b) superimposed on the current image.

[0064] The matching information is a vector that contains information on the target feature point and information on the current feature point corresponding to the target feature point. In matching, if SIFT is used as the feature, for example, a calculation method can be used in which the feature vectors for all combinations of the candidate feature point FP2 and the target feature point F100 are compared to find the current feature point corresponding to each target feature point.

[0065] For example, when N SIFT feature points are used as target feature points, the matching information (vector) includes information on the target feature points (u1d, v1d, z1d, u2d, v2d, z2d, uNd, vNd, zNd) and information on the current feature point (u1c, v1c, z1c, u2c, v2c, z2c, uNc, vNc, zNc). Here, uhd, vhd, and zhd respectively indicate the x-coordinate, y-coordinate, and z-coordinate of the h-th target feature point, and uhc, vhc, and zhc respectively indicate the x-coordinate, y-coordinate, and z-coordinate of the current feature point corresponding to the h-th target feature point (1≦h≦N). However, if the z-coordinate of the current feature point is not to be estimated (evaluated), that information need not be included.

[0066] In matching, if a corresponding current feature point cannot be detected among the candidate feature points for a certain target feature point, that target feature point can be excluded from the matching information. As already described, in this embodiment, the number of target feature points is set to exceed the minimum number of feature points L. The minimum number of feature points L is the number of feature points required so that the total degree of freedom of the information possessed by each feature point is equal to or greater than the degree of freedom required for visual servo control. Therefore, in this embodiment, a number of target feature points is set that ensures redundancy. This makes it possible to obtain information about the degree of freedom required for visual servo control without using all target feature points. FIG. 6(d) shows an example of the results of matching between the target feature point F100 in FIG. 5(c) and the candidate feature point FP2 in FIG. 6(b). The correspondence between the target feature point and the current feature point is indicated by arrows, and two target feature points for which a corresponding current feature point cannot be detected among the candidate feature points are indicated as target feature points F102. The two target feature points F102 for which a current feature point cannot be detected are excluded from the matching information.

[0067] The results of the matching process may be displayed on the display screen of the output device 103 as needed. FIG. 4(b) is an example of a screen when the matching results are displayed on the output device 103. By displaying the current feature points and target feature points together with the current image and target image on the image data display unit V100, the user can confirm whether the feature points have been matched correctly. As in the illustrated example, by also displaying arrows or the like indicating the correspondence between the current feature points and the target teaching points, the user can easily confirm whether the feature points have been matched correctly. Another suitable display method is to display the target feature points F102 for which no corresponding current feature points could be detected in a way that allows them to be identified.

[0068] Next, in step S24, an image Jacobian is generated from the matching information acquired in step S23. The image Jacobian is a matrix Ji that satisfies formula (1), where e' is the velocity of the coordinates of the current feature point on the image, and v is the minute movement of the image capture device 3.

number

[0069] In the classical six-degree-of-freedom image-based visual servoing using feature point coordinates, if e' is set as (du1c / dt, dv1c / dt, du2c / dt, dv2c / dt, . . .), the matrix Ji as the image Jacobian is set as shown in Equation (2), where f is the focal length of the imaging device 3.

number

[0070] In the above-mentioned step S23, it is not always possible to detect the current feature point for all target feature points. If there is a target feature point that could not be detected, the image Jacobian is generated excluding that target feature point. For example, if N=5 and the fourth target feature point is not detected, the image Jacobian will be as shown in Equation (3).

number

[0071] A known method can be used to set the image Jacobian depending on the system design. For example, the coordinates of the target feature point may be used instead of the coordinates of the current feature point. Alternatively, the average of the image Jacobian calculated using the target feature point and the image Jacobian calculated using the current feature point may be used, or an image Jacobian corresponding to only movement in some directions of the image capture device 3 may be used. Furthermore, if the z coordinate of the current feature point cannot be estimated, the image Jacobian may be calculated using the x and y coordinates of the current coordinates and the z coordinate of the target feature point.

[0072] Furthermore, when visual servo control is performed in a system in which the imaging device 3 does not move along with the robot 1, i.e., in a system in which images of the robot and the target object are captured using an imaging device in a fixed position, the image Jacobian can be modified appropriately.

[0073] Next, in step S25, the matching information generated in step S23 and the image Jacobian generated in step S24 are used to calculate the amount of feedback to the robot 1. In classical image-based visual servoing, the motion vg of the imaging device 3 controlled by the movement of the robot 1 is calculated according to equation (4).

number

[0074] Here, e is the difference between the current feature point and the target feature point (u1c-u1d, v1c-v1d, u2c-u2d, v2c-v2d, ...), Ji+ is the pseudo-inverse matrix of the image Jacobian, and λ is the feedback gain. It is desirable that the feedback gain is recorded in advance in the storage device 104 or the main storage device (RAM) of the control device 101 and read out when the calculation process is executed. Generally, the feedback gain λ is set to a value smaller than 1, and more preferably a small value around 0.1. The method of calculating vg using equation (4) is merely an example, and other known methods can be freely used.

[0075] In step S26, the robot control unit 108 converts the feedback amount calculated in step S25 into a control signal that can be interpreted by the robot 1 and transmits it to the robot 1. For example, if the robot 1 can interpret the motion vg of the image capture device 3 as a control signal, the calculation result of step S25 can be transmitted directly to the robot 1. If the control signal of the robot 1 is the joint angular velocity of each joint of the robot 1, the motion vg of the image capture device 3 is converted into the joint angular velocity of each joint of the robot 1 using a robot Jacobian or the like and transmitted to the robot 1. In addition, the control amount may be corrected using PID control or the like so that the robot 1 operates smoothly. Through this processing, the robot control unit 108 can operate the robot 1 so as to reduce the difference between the current feature points and the target feature points. In other words, the position and posture of the robot 1 can be controlled in a direction that matches the target image with the current image.

[0076] Next, in step S27, it is determined whether the robot 1 has reached a position and orientation that is within the allowable error range for the current target position set during a series of work operations. For example, if the feedback amount calculated in step S25 exceeds a predetermined range, it is determined that the position and orientation of the robot 1 have not yet converged to near the target position (step S27: NO). In that case, the process returns to step S21, and the processing from acquisition of the current image onwards is executed again. By quickly repeating the loop processing from step S21 to (step S27: NO), it is possible to converge, for example, the relative position between the workpiece 4 and the end effector 2 of the robot 1 to the target positional relationship.

[0077] Depending on the content of the work operation, the required positional accuracy may differ depending on the target position. For example, in an operation in which the end effector 2 approaches the vicinity of the workpiece 4, shortening the movement time may be required rather than high positional accuracy, and in an operation in which the end effector 2 contacts the workpiece 4, high positional accuracy may be required. In such cases, the control program can be configured so that the judgment criteria in step S27 can be changed appropriately depending on the target position.

[0078] Incidentally, if the position and posture of the robot 1 do not converge to the target position even after executing the loop process from step S21 (step S27: NO) multiple times, there is a possibility that some abnormality, such as a mechanical malfunction, has occurred in the robot 1. Therefore, the control program may be configured so that the robot control unit 108 counts the number of times the loop process is executed, and if the determination result in step S27 is NO even after the predetermined number of times has been reached, the control program proceeds to error processing. As error processing, the control program may be configured to, for example, stop the robot 1, display an error notification on the display screen of the output device 103, sound an alarm or issue a warning light, or send an error notification email to the user.

[0079] On the other hand, if the feedback amount calculated in step S25 is equal to or less than the predetermined amount, it is determined that the position and posture of the robot 1 have reached a position sufficiently close to the target position (step S27: YES), and the process proceeds to step S28.

[0080] In step S28, it is determined whether the series of operations of the robot 1 to be controlled by visual servo control has been completed. In other words, it is determined whether the visual servo control process has been completed for all of the series of target positions set according to the content of the work operation.

[0081] If there is a target position for which visual servo control has not been completed (step S28: NO), the process proceeds to step S29, where information on the target feature point linked to the next target position is read from the storage device, and the processes from step 21 onwards are executed again. When the processes for all target positions have been completed (step S28: YES), the visual servo control ends.

[0082] (Advantages of embodiment 1) The advantages of this embodiment will be explained in comparison with conventional visual servo control. In the following explanation, the number of coordinate axes for feedback control in the task space is six, and local features such as SIFT are used as image features.

[0083] An image processing method performed in conventional visual servo control will be described with reference to Figs. 7(a) to 7(d). Fig. 7(a) is an example of a target image captured by a camera, and corresponds to Fig. 5(a) referred to in the description of this embodiment. Fig. 7(b) shows target feature points F100 extracted from the target image by a conventional method. In this embodiment described with reference to Fig. 5(c), for example, seven target feature points F100 are extracted so that the number is greater than the minimum number of feature points L. In contrast, with the conventional method, as illustrated in Fig. 7(b), target feature points F100 that are less than or equal to the minimum number of feature points L (for example, four) are extracted.

[0084] Fig. 7(c) shows a current feature point F101 extracted from a current image by a conventional method. For ease of explanation, Fig. 7(d) uses arrows to indicate the correspondence between the current feature point F101 and the target feature point. Note that Figs. 7(a) to 7(d) are merely diagrams for convenience of explanation of conventional visual servo control, and do not necessarily mean that they are displayed on the screen of an output device.

[0085] In the conventional method, as shown in Figure 7(b), a sufficient number of target feature points are not extracted to ensure sufficient redundancy. Therefore, to ensure the required number of current feature points for feedback control, all candidate feature points extracted from the captured current image must be successfully matched with the target feature point F100, as shown in Figure 7(c). However, if the target object does not have features that are easily identifiable in the image (e.g., markers), it may be difficult to extract candidate feature points corresponding to target feature points from the current image depending on the object's position and orientation. For example, if it is not possible to identify the corresponding current feature point for even one of the target feature points shown in Figure 7(b), it will be impossible to ensure the required number of current feature points for proper feedback control. If any of the four extracted current feature points are erroneously detected, proper feedback control will also be impossible.

[0086] On the other hand, in this embodiment, as shown in Fig. 5(c), target feature points F100 are extracted with redundancy so that the number of target feature points F100 is greater than the minimum number of feature points L. Even if there is a target feature point F102 for which no corresponding candidate feature point FP2 can be detected in step S23, where feature point matching processing is performed, it is easy to ensure four or more current feature points, as shown in Fig. 6(d), and visual servo control can be performed appropriately.

[0087] That is, according to this embodiment, the information required to properly perform visual servoing can be stably acquired from the captured image, and visual servoing can be performed with higher reliability than ever before.

[0088] [Embodiment 2] 8 is a schematic diagram showing a schematic configuration of a robot system according to embodiment 2. The robot system includes a robot 1 and a controller 100 as a control unit. Descriptions of configurations and processes common to embodiment 1 will be simplified or omitted.

[0089] The robot 1 is, for example, a multi-joint robot with multi-axis control, but may also be any other type of robot or movable device. For example, the robot 1 may be a device with a movable part that can perform movements such as expansion and contraction, bending and stretching, vertical movement, horizontal movement, or rotation, or a combination of these movements.

[0090] In this embodiment, the control device 101 includes an image acquisition unit 105, a feature point extraction unit 106, a feature matching unit 107, and a robot control unit 108, as well as a feature point clustering unit 109. The feature point clustering unit 109 has a function of dividing the target feature points and current feature points extracted by the feature point extraction unit 106 into multiple clusters and calculating representative information for each cluster.

[0091] The visual servo control in the robot system according to this embodiment will now be described. Fig. 9 is a flowchart showing the flow of processing for moving the robot 1 to a target position in advance, capturing an image with the imaging device 3, and extracting target feature points, target feature point clusters, and target feature point cluster representative information based on the captured image, in preparation for implementing visual servo control. Fig. 10 is a flowchart showing the flow of processing for controlling the operation of the robot 1 by visual servo control. These steps will be explained in order below. (Preparing for visual servo control)

[0092] 9, the processes in steps S10, S11, S12, S15, and S16 are the same as those described with the same reference numerals in embodiment 1. Fig. 12(a) shows an example of the target image acquired in step S11. Fig. 12(b) schematically shows the feature point FP1 extracted in step S12 superimposed on the target image.

[0093] In step S90 of this embodiment, N (N is a plural number) target feature points can be selected from the feature points extracted from the target image in step S12 so that formula (5) is satisfied.

number

[0094] Here, n is the number of coordinate axes for which feedback control is performed in the task space, φ is the degree of freedom of information possessed by one feature cluster, k is the number of clusters, ψ is the degree of freedom of information possessed by each feature point, and N is the number of target feature points selected by the feature point extraction unit 106. φ and n are preset values, and are input by a user or loaded from a configuration file or program stored in the storage device 104 or main storage device (RAM) before execution of step S90. ψ, the degree of freedom of information possessed by one target feature point, is a value determined by the image features used in this embodiment and may be automatically calculated depending on the type of image feature, or may be preset in the same way as the degree of freedom of information possessed by each target feature point cluster φ and the number of degrees of freedom n for feedback control. The values ​​of the number k of target feature point clusters, N, may be automatically calculated to satisfy n+1≦φk≦ψN / 2, or may be preset in the same way as φ and n. It is preferable to satisfy at least one of n+1≦φk, φk≦ψN / 2, and n+1≦ψN / 2. n+1≦φk satisfies k≧(n+1) / φ, which is a similar relationship to N>(n+1) / ψ in embodiment 1. It is preferable to satisfy k>(n+1) / φ. φk≦ψN / 2 satisfies N≧2kφ / ψ, and if φ=ψ, then N≧2k. If N≧2k is satisfied, each target feature point cluster can be composed of multiple target feature points. It is preferable to satisfy N>2kφ / ψ, i.e., φk<ψN / 2. n+1≦ψN / 2 satisfies N≧2×(n+1) / ψ. Since L≧(n+1) / ψ, then N≧2×L. It is preferable that N>2×L, and it is preferable to satisfy N>2×(n+1) / ψ, i.e., n+1<ψN / 2.

[0095] The image features that can be used as feature points and the degree of freedom of information that each image feature has are explained in the same way as in step S13 in embodiment 1. For example, when φ=2, k=4, and SIFT is used as the image features, ψ=2, so the number N of feature points to be extracted is 8 or more.

[0096] 12(c) shows a schematic diagram of the target feature points F100 extracted in step S90 superimposed on the target image. The extracted target feature points are stored in the storage device 104 or the main storage device (RAM) so that they can be referenced in subsequent processing.

[0097] In step S91, the feature point clustering unit 109 performs a first clustering process. That is, the N target feature points selected in step S90 are divided into k clusters (target feature point clusters) (k is plural) so that Equation (5) is satisfied. For example, when φ = 2 and n = 6, k is 4 or greater. Any known clustering method can be used. For example, by dividing the feature points based on their image coordinates using a clustering method such as hierarchical cluster analysis using a nearest neighbor method or the k-means method, physically close feature points can be included in the same cluster. That is, clustering can also be performed based on the distance in the target image. Alternatively, by dividing the feature points into superpixels using the Slic method or Watershed method and classifying feature points within the same superpixel into the same cluster, feature points on the same surface can be more easily grouped into the same cluster, which is expected to improve the accuracy of calculations such as affine transformation, which will be described later.

[0098] Clustering may be performed based on the brightness values ​​(gradation levels) of target feature points in the target image. Furthermore, if the workpiece 4 is divided based on a shape model so that feature points on the same surface are included in the same target feature point cluster, the accuracy of calculations such as affine transformation can be improved more reliably than with the above-mentioned method. More preferably, if the division is performed so that each cluster contains two or more target feature points, the target feature point cluster can be detected even if some of the feature points included in each target feature point cluster are not detected. Furthermore, if the target image includes both the workpiece and the robot, such as when the imaging device is installed outside the robot, clustering may be performed based on the shape model of the robot as well as the shape model of the workpiece.

[0099] 12(d) schematically shows the clustering results obtained in step S91 superimposed on the target image. In the example shown, the target feature points selected in step S90 are divided (classified) into four target feature point clusters: F110, F120, F130, and F140. In the figure, the target feature points that make up each target feature point cluster are indicated by different marks for each target feature point cluster.

[0100] Next, in step S92, the feature point clustering unit 109 calculates target feature point cluster representative information for each of the k target feature point clusters divided in step S91. The target feature point cluster representative information is information that represents the target feature point cluster as a single image feature, and is used later when performing feedback control. For example, the center of gravity position of multiple target feature points included in the target feature point cluster can be used as the target feature point cluster representative information, in which case φ=2. By performing feedback control using the cluster representative information, it is possible to suppress changes in the accuracy of the control information to be fed back, even if the target feature points that make up the target feature point cluster change. The target feature point cluster representative information for a certain target feature point cluster may be any of the target feature points included in that target feature point cluster.

[0101] Note that the target feature point cluster representative information may include, in addition to the centroid position of the target feature point, a magnification rate and / or a rotation angle that can be calculated by calculation such as affine transformation (described later), in which case φ=3 or φ=4. The processing result may be displayed on the output device 103 as needed.

[0102] 12(e) schematically shows the results calculated in step S92 superimposed on the target image. Target feature point cluster representative information F115, target feature point cluster representative information F125, target feature point cluster representative information F135, and target feature point cluster representative information F145 are schematically shown with different marks.

[0103] In step S93, the target feature point cluster information and the target feature point cluster representative information are stored in the storage device 104 or the main storage device (RAM) so that they can be referenced in subsequent processing.

[0104] 11(a) is an example of a display screen displayed on the output device 103 in steps S11, S12, S90, S91, S92, and S93 of this embodiment. The display screen may include an image data display section V100, a capture button V101, a registration button V102, and a parameter input field V103. The parameter input field V103 can be used by the user to set parameters such as n, φ, and ψ using the input device 102.

[0105] The capture button V101 and the registration button V102 can be operated by the user using the input device 102. By turning on the capture button V101, the processing from step S11 to step S92 is started. In step S11, a target image as shown in FIG. 12(a), for example, is displayed on the image data display unit V100, and in step S12, candidate feature points as shown in FIG. 12(b), for example, are displayed on the image data display unit V100. In step S90, target feature points as shown in FIG. 12(c), for example, are displayed on the image data display unit V100. In addition, in step S91, a clustering result as shown in FIG. 12(d), for example, is displayed, and in step S92, cluster representative information as shown in FIG. 12(e), for example, is displayed on the image data display unit V100.

[0106] On each display screen, the target feature points and cluster representative information may be displayed in different shapes or colors so that it is easy to distinguish which cluster they belong to. The user may select the registration button V102 to record the processing results as the target image and target feature points and proceed to the next step, or if there is a problem with the processing results, may select the capture button V101 again to execute the processing from step S11 onwards again.

[0107] Note that some or all of the screen displays described above may not be implemented, and the screen design may be changed as appropriate. For example, the functions of the capture button V101 and the registration button V102 may be divided into more buttons, or their functions may be realized by command input. Also, the parameter input field V103 is not necessarily required; it may be written in a separate file in advance, or may be set by command input. (Visual Servo Control)

[0108] With reference to Fig. 10, a procedure for causing the robot 1 to perform a task using visual servo control in this embodiment will be described. The processes in steps S20 to S22 shown in Fig. 10 are the same as those described in Fig. 3 of the first embodiment using the same reference numerals. Fig. 13(a) shows an example of the current image acquired in step S21, and Fig. 13(b) shows an example in which the candidate feature point FP2 extracted in step S22 is superimposed on the current image. The image shown in Fig. 13(a) or 13(b) can be displayed on the display screen of the output device 103.

[0109] In step S23 of this embodiment, matching information is generated by performing a matching process between candidate feature points and target feature points, as in embodiment 1. In the following description, for convenience, the candidate feature points that can be associated with target feature points through matching are referred to as "current feature points."

[0110] In matching, for example, when SIFT is used as a feature, a method can be used in which the feature vectors for all combinations of candidate feature point FP2 and target feature point F100 are compared to find the current feature point corresponding to each target feature point.

[0111] For example, when N SIFT feature points are used as target feature points, the matching information (vector) includes information on the target feature points (u1d, v1d, z1d, u2d, v2d, z2d..., uNd, vNd, zNd) and information on the current feature point (u1c, v1c, z1c, u2c, v2c, z2c..., uNc, vNc, zNc). Here, uhd, vhd, zhd (h is 1 to N) respectively indicate the x-coordinate, y-coordinate, and z-coordinate of the h-th target feature point, and uhc, vhc, zhc respectively indicate the x-coordinate, y-coordinate, and z-coordinate of the current feature point corresponding to the h-th target feature point. However, if the z-coordinate of the current feature point is not to be estimated, that information need not be included.

[0112] In matching, if a corresponding current feature point cannot be detected from among the candidate feature points for a certain target feature point, that target feature point can be excluded from the matching information. As already explained, in this embodiment, the number of target feature points is set so that Equation (5) is satisfied, and therefore, a number of target feature points is set that ensures redundancy. Therefore, information about the degrees of freedom required for visual servo control can be obtained without using all target feature points.

[0113] Figure 13(c) shows a schematic diagram of the matching process between target feature points and current feature points. The correspondence between target feature points and current feature points is indicated by arrows, and target feature points for which no corresponding current feature point can be detected among the candidate feature points are indicated by isolated reference symbols without arrows. Target feature points for which no corresponding current feature point can be detected are excluded from the matching information.

[0114] Fig. 13(d) is a schematic diagram showing candidate feature points that have been associated with target feature points through matching, i.e., current feature points, superimposed on the current image. The images shown in Fig. 13(c) or 13(d) can be displayed on the display screen of the output device 103. If, as in the example shown in Fig. 13(c), symbols such as arrows indicating the association between the current feature points and the target feature points are also displayed, the user can easily confirm whether the feature points have been matched correctly. Another suitable display method is to display the current feature points in a way that allows them to distinguish target feature points that could not be detected from the candidate feature points.

[0115] In this embodiment, once the current feature point is extracted in step S23, a process of estimating (calculating) cluster representative information for the current feature point is performed in step S104.

[0116] In step 104, first, as a second clustering process, the current feature points are divided (classified) into a plurality of current feature point clusters. FIG. 13(e) schematically shows the result of dividing (classifying) the current feature points into a plurality of current feature point clusters, superimposed on the current image. The feature point clustering unit 109 divides (classifies) the current feature points shown in FIG. 13(d) into a plurality of clusters, corresponding to the cluster division (classification) of the target feature points shown in FIG. 12(d). That is, if a target feature point associated by matching belongs to target feature point cluster F110, the current feature point is classified into current feature point cluster F111 corresponding to target feature point cluster F110. Similarly, if a target feature point associated by matching belongs to target feature point cluster F120, target feature point cluster F130, or target feature point cluster F140, the corresponding current feature points are classified into current feature point cluster F121, current feature point cluster F131, and current feature point cluster F141, respectively. (The corresponding target feature cluster and current feature cluster are shown in the same order.)

[0117] If each current feature point cluster contains three or more current feature points, it is possible to determine the affine transformation between the target feature points and the current feature points. Furthermore, although there is a possibility that incorrect candidate feature points have been extracted as current feature points due to mismatching, if there are three or more correctly detected feature points in a cluster, it is possible to remove the incorrect current feature points and perform appropriate affine transformation by using a robust estimation method such as RANSAC. In this way, the number N of target feature points extracted from the target image should preferably satisfy N≧3×L, where M is the smallest integer that satisfies M>(n+1) / ψ.

[0118] The feature point clustering unit 109 may perform clustering of the current feature points based on the distance in the current image. Alternatively, the current feature points may be clustered based on the brightness value (grayscale level) in the current image. Alternatively, the current feature points on the same surface may be clustered based on the shape model of the workpiece 4 so that they are included in the same current feature point cluster. Furthermore, if the current image includes both the workpiece and the robot, such as when the imaging device is installed outside the robot, the clustering may be performed based on the shape model of the workpiece and the shape model of the robot.

[0119] Next, the feature point clustering unit 109 calculates current feature point cluster representative information for each current feature point cluster. The current feature point cluster representative information is information that represents the current feature point cluster as a single image feature, and is used later when performing feedback control. The current feature point cluster representative information based on the current image is obtained by performing a process similar to that used to obtain the target feature point cluster representative information in step S92. However, a calculation method different from that used in step S92 may be used when calculating the current feature point cluster representative information. The current feature point cluster representative information for a certain current feature point cluster may be any of the current feature points included in that current feature point cluster.

[0120] Figure 13(f) is a schematic diagram showing the acquired current feature point cluster representative information superimposed on the current image. As can be seen by comparing with Figure 13(e), current feature point cluster representative information F116 has been calculated for the current feature point cluster F111, and current feature point cluster representative information F126 has been calculated for the current feature point cluster F121. Similarly, current feature point cluster representative information F136 has been calculated for the current feature point cluster F131, and current feature point cluster representative information F146 has been calculated for the current feature point cluster F141.

[0121] When the processing of step S104 is executed, the image exemplified in Fig. 13(e) or 13(f) can be displayed on the display screen of the output device 103. The acquired current feature point cluster representative information is stored in the storage device 104 or the main storage device (RAM) so as to be used in feedback control.

[0122] Next, in step S24, an image Jacobian is generated. Explanation of the image Jacobian is omitted here, as it is basically the same as that described in embodiment 1. In this embodiment, as schematically shown in Fig. 14, target feature point cluster representative information and current feature point cluster representative information are used as matching information.

[0123] Next, in step S25, the matching information and the image Jacobian generated in step S24 are used to calculate the amount of feedback for the robot 1. The explanation of the method for calculating the amount of feedback is omitted here, as it is basically the same as that explained in the first embodiment. Explanation of steps S26 to S29 is omitted because they are basically the same as those explained using the same reference numerals in the first embodiment.

[0124] With the above processing, even if current feature points corresponding to some target feature points cannot be detected, as long as a certain number or more of current feature points are detected in each current feature point cluster, it is possible to estimate current feature point cluster representative information and perform appropriate visual servo control.

[0125] Note that the formula (5) in the above explanation can be changed to a stricter condition depending on the method for estimating the current feature point cluster representative information in step S104. For example, the formula (5) can be expanded to the formula (6).

number

[0126] Here, m is a natural number equal to or greater than 2. If formula (6) is satisfied, each cluster can contain m or more feature points through cluster division, and therefore, by appropriately selecting m, it is possible to estimate cluster representative information with higher accuracy. For example, if m = 3, each cluster will contain three or more feature points, and therefore it is possible to calculate the affine transformation between the current feature points and the target feature points. By performing the calculated affine transformation on the cluster representative information of the target image, it is possible to estimate the current value of the cluster representative information (current feature point cluster representative information) while suppressing the effects of fluctuations in the cluster center of gravity due to the feature point detection state, etc.

[0127] Furthermore, if m is set to 4 or more, cluster representative information can be estimated even if some feature points are not detected for any cluster. In addition, using methods such as RANSAC and LMedS, cluster representative information can be estimated robustly, making it possible to deal with not only undetected feature points but also erroneous detections.

[0128] The matching results may be displayed on the output device 103 as needed. Fig. 11(b) is an example of a display screen when the matching results are displayed on the output device 103. By displaying the current image and the current feature point cluster representative information F116, F126, F136, and F146 superimposed on the image data display unit V100, the user can confirm whether the current feature point cluster representative information has been estimated correctly. Another preferred display method is to display the target image and the target feature point cluster representative information F115, F125, F135, and F145 superimposed on each other.

[0129] Furthermore, a preferred display method is to add a line indicating the correspondence between the current feature point cluster representative information and the target feature point cluster representative information. Furthermore, the target feature point clusters F110, F120, F130, and F140 and the current feature point clusters F111, F121, F131, and F141 may be displayed, or the results of RANSAC or LMedS may be displayed. This allows the user to check the processing results in more detail. The above feature points and cluster representative information may be displayed in different shapes or colors so that it is possible to distinguish which cluster they belong to.

[0130] (Advantages of embodiment 2) In the control method according to the first embodiment, target feature points are extracted with redundancy so that the number of target feature points is greater than the minimum number L of feature points. Therefore, even if there is a target feature point F102 for which a corresponding current feature point cannot be detected, visual servo control can be performed with higher accuracy than conventional methods. However, for example, if imaging conditions such as external light are prone to change when capturing a current image, the target feature point for which matching is successful (see FIG. 6(d)) may not always be the same target feature point. In other words, the target feature points used to calculate the control amount for visual servo control may change (be replaced) each time a current image is captured. Generally, the tendency for errors to occur in the feature point detection process differs for each feature point. Therefore, in the first embodiment, different feature points may be detected each time a control loop is executed in visual servo control. If the target feature points used for control are not consistent depending on the control timing, the error in the feedback signal may change each time, making it difficult to control the robot quickly and smoothly. For example, it may take a long time for the control results to converge, or the control accuracy may fluctuate over time.

[0131] In the second embodiment, as in the first embodiment, target feature points are extracted with redundancy so that the number of target feature points is greater than the minimum number L of feature points. This makes it possible to stably obtain information necessary for properly performing visual servoing from captured images, thereby enabling visual servoing to be performed with higher reliability than in the past. Furthermore, in the second embodiment, target feature points are clustered to obtain target feature point cluster representative information for each cluster, and current feature points are also clustered to obtain current feature point cluster representative information. Then, the target feature point cluster representative information extracted from the target image and the current feature point cluster representative information extracted from the current image are used to calculate the amount of feedback and perform visual servoing control. For example, even if the imaging conditions change each time the current image is captured and the target feature points that result in successful matching fluctuate (are replaced), the cluster representative information is less susceptible to this influence, enabling feedback control that is even more stable than in the first embodiment.

[0132] In this embodiment, feature point cluster representative information estimated from multiple feature points included in a cluster is used to calculate the feedback amount, so the tendency for errors in the feedback signal to occur is less likely to change each time a control loop is executed, which has the advantage of making it easier to control the robot faster and more smoothly than in embodiment 1.

[0133] [Embodiment 3] 15 is a schematic diagram showing a schematic configuration of a robot system according to embodiment 3. The robot system includes a robot 1 and a controller 100 as a control unit. Descriptions of configurations and processes common to embodiment 1 will be simplified or omitted.

[0134] The robot 1 is, for example, a multi-joint robot with multi-axis control, but may also be any other type of robot or movable device. For example, the robot 1 may be a device with a movable part that can perform movements such as expansion and contraction, bending and stretching, vertical movement, horizontal movement, or rotation, or a combination of these movements.

[0135] In this embodiment, the control device 101 includes an image acquisition unit 105, a feature point extraction unit 106, a feature matching unit 107, and a robot control unit 108, as well as a feature priority setting unit 110 and a feature complementation unit 111. The feature priority setting unit 110 has a function of setting priorities for the target feature points extracted by the feature point extraction unit 106, and the feature complementation unit 111 has a function of deleting and adding current feature points based on the matching information generated by the feature matching unit 107 and the priorities set by the feature priority setting unit 110, to generate complemented current feature points.

[0136] Note that "complementation" here means correcting extracted current feature points, and is not limited to supplementing missing current feature points, but also includes deleting current feature points with low priority, or changing one current feature point to another. Therefore, "complementation processing" can also be rephrased as correction processing.

[0137] The visual servo control in the robot system according to this embodiment will now be described. Fig. 16 is a flowchart showing the process flow for moving the robot 1 to a target position in advance, capturing an image with the imaging device 3, extracting target feature points based on the captured image, and setting priorities in preparation for visual servo control. Fig. 17 is a flowchart showing the process flow for controlling the operation of the robot 1 by visual servo control. These processes will be explained in order below.

[0138] (Preparing for visual servo control) In Fig. 16, the processes of steps S10, S11, S12, S13, S15, and S16 are the same as those described in embodiment 1 using the same reference numerals. Fig. 19(a) shows an example of the target image acquired in step S11. Fig. 19(b) schematically shows the feature point FP1 extracted in step S12 superimposed on the target image. Fig. 19(c) schematically shows the target feature point F100 extracted in step S13 superimposed on the target image.

[0139] In step S161, the feature priority setting unit 110 performs a feature priority setting process. That is, priorities to be used in the feedback process are set for the N target feature points extracted in step S13. The priorities may be set manually by a user or may be automatically determined according to the evaluation values ​​of each feature point.

[0140] Here, the evaluation value of each feature point numerically represents whether the use of that feature point in generating a feedback signal will improve the system's performance and reliability. Any known method can be used to calculate the evaluation value. For example, steps S11, S12, and S13 may be performed multiple times, and each feature point may be weighted according to the number of times it is extracted (detection frequency), and the weighted evaluation value may be used. Similarly, the detection accuracy when the same feature point is extracted multiple times, i.e., the standard deviation of the coordinates, may also be used. When performing steps S11, S12, and S13 multiple times, the image capture device 3 may be placed in multiple different positions, and a group of images captured from the multiple positions may be acquired. In this case, the detection accuracy may be evaluated using, for example, the difference between the predicted position of each feature point calculated using the image Jacobian and the actual extracted position.

[0141] Alternatively, the contribution of each feature point to visual servo control performance can be used as the evaluation value. For example, when L is the smallest integer M that satisfies M>(n+1) / ψ, L target feature points can be selected from N target feature points, and the sensitivity when calculating the image Jacobian using only the selected target feature points, i.e., the minimum singular value of the image Jacobian, can be calculated. The sensitivity can be calculated for all combinations of the L target feature points, and for each feature point, the sum of the sensitivities of the combinations that include that feature point can be used as the evaluation value.

[0142] 19(d) schematically shows the priorities set for each target feature point in step S161, superimposed on the target image. In the example shown, seven target feature points F100 are assigned ranks 1 to 7. The smaller the number, the higher the priority. Next, in step S162, the target feature points and priorities are stored in the storage device 104 or the main storage device (RAM) so that they can be referenced in later processing.

[0143] 18(a) is an example of a display screen displayed on the output device 103 in steps S11, S12, S13, S161, and S162 of this embodiment. The display screen may include an image data display section V100, a capture button V101, and a registration button V102.

[0144] The user can operate the capture button V101 and the registration button V102 using the input device 102. Turning on the capture button V101 starts the processing from step S11 to step S161. In step S11, a target image as shown in FIG. 19(a), for example, is displayed on the image data display unit V100, and in step S12, candidate feature points as shown in FIG. 19(b), for example, are displayed on the image data display unit V100. In step S13, target feature points as shown in FIG. 19(c), for example, are displayed on the image data display unit V100. In addition, in step S161, the result of feature point prioritization is displayed, as shown in FIG. 19(d), for example. When the priority is set manually, a function for setting the priority may be provided, for example, by the user specifying the target feature points and the priority on the image data display unit V100 using the input device 102.

[0145] On each display screen, the target feature points and their priorities may be displayed near each target feature point, or may be displayed in a different color. The user may select the registration button V102 to record the processing results as the target image and target feature points and proceed to the next step, or if there is a problem with the processing results, may select the capture button V101 again to execute the processing from step S11 onwards again.

[0146] Note that some or all of the above-described screen displays may not be implemented, and the screen design may be changed as appropriate. For example, the functions of the capture button V101 and the registration button V102 may be divided into more buttons, or the functions may be realized by command input.

[0147] (Visual Servo Control) With reference to FIG. 17, a procedure for causing the robot 1 to perform a task using visual servo control in this embodiment will be described. The processes in steps S20 to S23 shown in FIG. 17 are the same as those described in FIG. 3 of the first embodiment using the same reference numerals. FIG. 20(a) shows an example of the current image acquired in step S21, and FIG. 20(b) shows an example of the candidate feature point FP2 extracted in step S22 superimposed on the current image. FIG. 20(c) is a schematic diagram showing the candidate feature point that has been associated with the target feature point in step S23, i.e., the current feature point, superimposed on the current image. The images shown in FIG. 20(a), 20(b), or 20(c) can be displayed on the display screen of the output device 103.

[0148] In this embodiment, once current feature points are extracted in step S23, a process of deleting or adding current feature points to generate complemented current feature points is performed in step S171. In step S171, first, the priority of the target feature points associated with the current feature points is confirmed. Hereinafter, a current feature point associated with a target feature point of priority M will be referred to as a current feature point of priority M. If all current feature points of priorities 1 to L have been detected with respect to the minimum number of feature points L, then the current feature points obtained by deleting those with priorities lower than L (those with priority numbers higher than L) are set as complemented current feature points. In other words, the number of complemented current feature points matches the minimum number of feature points L.

[0149] On the other hand, if there is an undetected current feature point among the priorities 1 to L, the current feature point is complemented using other information to generate a complemented current feature point. The simplest method for complementing current feature points is to use current feature points with a priority lower than L. For example, suppose L=4 and current feature points with priorities 1, 2, 4, 5, and 6 have been detected. Since the current feature point with priority 3 has not been detected, the current feature point with priority 5 is used instead, and the current feature point with priority 6 is deleted, thereby generating a number of complemented current feature points that matches the minimum number of feature points L.

[0150] Furthermore, as a method for complementing current feature points, undetected current feature points can be estimated and the estimated values ​​can be used instead of the current feature points. Any known method can be freely used to estimate undetected current feature points, and preferably, coordinate transformations (geometric transformations) such as Affine transformations and Homography transformations can be used, or the past movement history of current feature points can be used. For example, assume that L=4 and current feature points with priorities 1, 2, 4, 5, and 6 have been detected. In this case, it is possible to obtain Homography transformations between the current feature points with priorities 1, 2, 4, and 5 and the corresponding target feature points. By applying a similar transformation to the target feature point with priority 3, the position of the current feature point with priority 3 can be estimated.

[0151] Alternatively, if the positions of each current feature point, the amount of movement of the image capture device 3, and a physical model associating them are stored in a main memory device (RAM) or the like each time steps S23 and S25 are executed, the current feature point of priority 3 can be estimated using a Bayes filter such as a Kalman filter. By adding the estimated values ​​of the current feature points of priority 3 obtained by these methods to the current feature points and deleting the current feature points of priorities 5 and 6, it is possible to generate a number of complemented current feature points that matches the minimum number L of feature points.

[0152] The method for generating an interpolated current feature point (method for correcting an extracted current feature point) may be a combination of two or more of the above methods. For example, if the reliability of the estimated value of the current feature point with priority 3 estimated by the Kalman filter is low, the current feature point with priority 5 may be left as it is instead of adding the estimated value of the current feature point with priority 3.

[0153] Fig. 20(d) is a schematic diagram showing the processing results of step S171, i.e., the complemented current feature points and the deleted current feature points, superimposed on the current image. The image shown in Fig. 20(d) can be displayed on the display screen of the output device 103. As in the example shown in Fig. 20(d), a suitable display method is to display the estimated values ​​F103 of the added current feature points and the deleted current feature points F104 using different symbols or colors so that they can be distinguished from the detected current feature points.

[0154] Step S24: Generate an image Jacobian. The description of the image Jacobian is omitted here, as it is basically the same as that in the first embodiment. In this embodiment, the image Jacobian is generated using matching information between the interpolated current feature points and the target feature points.

[0155] Next, in step S25, the amount of feedback for the robot 1 is calculated using the matching information between the complemented current feature points and the target feature points, and the image Jacobian generated in step S24. The explanation of the method for calculating the amount of feedback is omitted because it is basically the same as the matters explained in embodiment 1. The explanation of steps S26 to S29 is omitted because it is basically the same as the matters explained using the same reference numerals in embodiment 1 (FIG. 3).

[0156] Note that, instead of the target feature points and current feature points described above, target feature point cluster representative information and current feature point cluster representative information may be used, as described in embodiment 2. As a first method, the current feature points and target feature points supplemented (corrected) based on the priority as described above may be classified into target feature point cluster representative information and current feature point cluster representative information, as described in embodiment 2, and an image Jacobian may be generated in step S24. As a second method, the target feature points may be pre-classified into more target feature point clusters than the minimum required number of clusters, and each target feature point cluster may be assigned a priority. If a current feature point cluster corresponding to a target feature point cluster with a high priority cannot be classified from the current feature points, or if classification is possible but is evaluated to have defects, a current feature point cluster corresponding to a target feature point cluster with a low priority can be used instead. That is, part of the current feature point cluster representative information corresponding to the target feature point cluster representative information with a high priority can be supplemented (corrected) using current feature point cluster representative information corresponding to the target feature point cluster representative information with a low priority. The necessary number of pieces of current feature point cluster representative information thus supplemented (corrected) are extracted in descending order of priority, and an image Jacobian is generated in step S24. Then, using the matching information between the target feature point cluster representative information and the current feature point cluster representative information, and the image Jacobian generated in step S24, the amount of feedback to the robot 1 can be calculated in step S25. Note that the description of the image Jacobian and steps S26 to S29 is omitted here, as they are basically the same as those described in the first embodiment.

[0157] (Advantages of embodiment 3) In the control method according to the first embodiment, target feature points are extracted with redundancy so that the number of target feature points is greater than the minimum number L of feature points. Therefore, even if there is a target feature point F102 for which a corresponding current feature point cannot be detected, visual servo control can be performed with higher accuracy than conventional methods. However, for example, if imaging conditions such as external light are prone to change when capturing a current image, the target feature point for which matching is successful (see FIG. 6(d)) may not always be the same target feature point. In other words, the target feature points used to calculate the control amount for visual servo control may change (be replaced) each time a current image is captured. Generally, the tendency for errors to occur in the feature point detection process differs for each feature point. Therefore, in the first embodiment, different feature points may be detected each time a control loop is executed in visual servo control. If the target feature points used for control are not consistent depending on the control timing, the error in the feedback signal may change each time, making it difficult to control the robot quickly and smoothly. For example, it may take a long time for the control results to converge, or the control accuracy may fluctuate over time.

[0158] In the third embodiment, as in the first embodiment, target feature points are extracted with redundancy so that the number of target feature points is greater than the minimum number L of feature points. This makes it possible to stably obtain information necessary for properly performing visual servoing from captured images, thereby enabling visual servoing processing to be performed with higher reliability than in the past. Furthermore, in the third embodiment, priorities are set for target feature points, and the same number of interpolated current feature points as the minimum number L of feature points are generated. An image Jacobian is then generated using the interpolated current feature points, and the amount of feedback is calculated using matching information between the interpolated current feature points and the target feature points, thereby performing visual servoing control. For example, even if the imaging conditions change each time a current image is captured and the target feature points that result in successful matching fluctuate (are replaced), the interpolated current feature points are less susceptible to this influence, enabling feedback control that is even more stable than in the first embodiment.

[0159] In this embodiment, the visual servo control performance is improved and the feedback amount is calculated by preferentially using feature points with high detection performance, so the tendency for errors in the feedback signal to occur is less likely to change with each control loop, which has the advantage that the robot can be controlled more quickly and smoothly than in embodiment 1.

[0160] [Other embodiments] The present invention is not limited to the above-described embodiments and examples, and many modifications are possible within the technical spirit of the present invention. For example, the above-described different embodiments may be combined. A control program that enables a computer to execute the above-described control method, and a computer-readable recording medium storing the control program are also included in the embodiments of the present invention.

[0161] The control method and system of the present invention can be applied to the control of various machines and equipment, including production equipment with moving parts, such as industrial robots, service robots, and machines that operate under computer numerical control.

[0162] The present invention can also be realized by supplying a program that realizes one or more functions of the embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. In the above-described embodiments, the various pieces of information described as being calculated or extracted can also be obtained by methods other than calculation or extraction. For example, the information can be obtained not only by calculation, but also by replacing complex calculation processing with a reference process to a simple array (lookup table (LUT)), or by using machine learning.

[0163] This specification discloses at least the following: [Matter 1] A control method for controlling a device having a moving part by visual servoing, comprising: acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points having a degree of freedom of information of ψ from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; generating a control signal for moving the movable part with n degrees of freedom using matching information based on the matching between the current feature points and the target feature points obtained by the matching process; The number N of the target feature points extracted from the target image satisfies N≧2×L, where L is the smallest integer M that satisfies M>(n+1) / ψ. A control method comprising: [Matter 2] classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; the matching information includes the current feature point cluster representative information and the target feature point cluster representative information; 2. The control method according to item 1, [Matter 3] A priority is set for each of the extracted target feature points; In the matching process, a current feature point that corresponds to the target feature point having a high priority is preferentially extracted. 3. The control method according to item 1 or 2, [Matter 4] a priority is set for each of the classified target feature point clusters; In the matching process, a current feature point cluster associated with the target feature point cluster having a higher priority is preferentially acquired. 3. The control method according to item 2. [Matter 5] A control method for controlling a device having a moving part by visual servoing, comprising: acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points having a degree of freedom of information of ψ from the target image by image processing; classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; generating a control signal for moving the movable part with n degrees of freedom using matching information between the current feature point cluster representative information and the target feature point cluster representative information; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; A control method comprising: [Matter 6] A control method for controlling a device having a moving part by visual servoing, comprising: acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points having a degree of freedom of information of ψ from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; correcting the current feature points using priorities set for the current feature points and the target feature points obtained by the matching process; generating a control signal for moving the movable part with n degrees of freedom using matching information between the corrected current feature points and the target feature points; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; A control method comprising: [Matter 7] N≧1+(n+1) / ψ 7. The control method according to item 5 or 6, [Matter 8] When L is the smallest integer M that satisfies M>(n+1) / ψ, N≧L+1 is satisfied. 7. The control method according to item 5 or 6, [Matter 9] N≧2×L is satisfied. 7. The control method according to item 5 or 6, [Matter 10] The number N of the target feature points extracted from the target image satisfies N≧3×L, where L is the smallest integer M that satisfies M>(n+1) / ψ. 7. The control method according to any one of items 1 to 6, [Matter 11] N≦L+40 is satisfied. 7. The control method according to any one of items 1 to 6, [Matter 12] N≦10×L 12. The control method according to item 11. [Matter 13] n is 6 or 7 and ψ is 2 or 3; 13. The control method according to item 12. [Matter 14] the number k of the plurality of target feature point clusters and the degree of freedom φ of information held by each target feature point cluster satisfy n+1≦φk≦ψN / 2, 3. The control method according to item 2. [Matter 15] in the second clustering process, the current feature points associated with the target feature points in the matching process are classified into the current feature point cluster corresponding to the target feature point cluster to which the target feature points belong; 3. The control method according to item 2. [Matter 16] In the first clustering process, the plurality of target feature points are classified into the plurality of target feature point clusters based on distances in the target image. and / or In the second clustering process, the plurality of current feature points are classified into the plurality of current feature point clusters based on distances in the current image. 3. The control method according to item 2. [Matter 17] In the first clustering process, the plurality of target feature points are classified into the plurality of target feature point clusters based on gradation levels in the target image. and / or In the second clustering process, the plurality of current feature points are classified into the plurality of current feature point clusters based on gradations in the current image. 3. The control method according to item 2. [Matter 18] in the first clustering process, classifying the plurality of target feature points into the plurality of target feature point clusters based on a shape model of the object and / or a shape model of the movable part; and / or in the second clustering process, the plurality of current feature points are classified into the plurality of current feature point clusters based on a shape model of the object and / or a shape model of the movable part; 3. The control method according to item 2. [Matter 19] acquiring, as the target feature point cluster representative information, a center of gravity position of the target feature points included in each of the plurality of target feature point clusters; acquiring, as the current feature point cluster representative information, a center of gravity position of the current feature points included in each of the plurality of current feature point clusters; 3. The control method according to item 2. [Matter 20] acquiring, as the target feature point cluster representative information, a center of gravity position and a magnification ratio of the target feature point included in each of the plurality of target feature point clusters, or the center of gravity position and a rotation angle, or the center of gravity position, the magnification ratio, and the rotation angle; acquiring, as the current feature point cluster representative information, the center of gravity position and magnification ratio of the current feature point included in each of the plurality of current feature point clusters, or the center of gravity position and rotation angle, or the center of gravity position, magnification ratio, and rotation angle; 3. The control method according to item 2. [Matter 21] the priority order is set based on the detection frequency of the target feature points; 7. The control method according to item 6, [Matter 22] the priority is set according to the contribution of the target feature point to visual servo control performance; 7. The control method according to item 6, [Matter 23] The priority order is set based on the detection accuracy of the target feature points. 7. The control method according to item 6, [Matter 24] The priority order is set using a group of images taken from a plurality of positions. 7. The control method according to item 6, [Matter 25] the modified current feature point is generated using the current feature point with the lower priority. 7. The control method according to item 6, [Matter 26] the modified current feature point is generated using another current feature point through a geometric transformation; 7. The control method according to item 6, [Matter 27] The modified current feature points are generated using a physical model and a motion history. 7. The control method according to item 6, [Matter 28] The image processing uses image features that are invariant to rotation and scaling. 28. A control method according to any one of items 1 to 27, characterized in that [Matter 29] the target image and the current image are captured by an imaging device attached to the movable part; 29. A control method according to any one of items 1 to 28, characterized in that [Matter 30] the target image and the current image are captured by an imaging device installed at a position where the movable part and the object can be imaged; 29. A control method according to any one of items 1 to 28, characterized in that [Matter 31] The device having the movable part is a robot. 31. The control method according to any one of items 1 to 30, [Matter 32] A control program for causing a control unit to execute the control method according to any one of items 1 to 31. [Matter 33] 33. A computer-readable recording medium having the control program according to item 32 recorded thereon. [Matter 34] The apparatus is an apparatus used to manufacture an article, and the apparatus is controlled by the control method according to any one of items 1 to 31 to manufacture the article. A method for manufacturing an article. [Matter 35] A system including a device having a movable part and a control unit, The control unit acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points having a degree of freedom of information of ψ from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; generating a control signal for moving the movable part with n degrees of freedom using matching information based on the matching between the current feature points and the target feature points obtained by the matching process; The number N of the target feature points extracted from the target image satisfies N≧2×L, where L is the smallest integer M that satisfies N>(n+1) / ψ. A system characterized by: [Matter 36] A system including a device having a movable part and a control unit, The control unit acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points having a degree of freedom of information of ψ from the target image by image processing; classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; generating a control signal for moving the movable part with n degrees of freedom using matching information between the current feature point cluster representative information and the target feature point cluster representative information; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; A system characterized by: [Matter 37] A system including a device having a movable part and a control unit, The control unit acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points having a degree of freedom of information of ψ from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; correcting the current feature points using priorities set for the current feature points and the target feature points obtained by the matching process; generating a control signal for moving the movable part with n degrees of freedom using matching information between the corrected current feature points and the target feature points; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; A system characterized by: [Matter 38] A priority is set for each of the extracted target feature points; In the matching process, a current feature point that corresponds to the target feature point having a high priority is preferentially extracted. 36. The system of claim 35. [Matter 39] a priority is set for each of the classified target feature point clusters; In the matching process, a current feature point cluster associated with the target feature point cluster having a higher priority is preferentially acquired. 37. The system of claim 36. [Matter 40] The device having a movable part is a robot. 40. The system of any one of items 35 to 39. [Matter 41] The control unit displaying on a display screen some or all of the image captured by the imaging device, the target image, the feature points, the target feature points, the current image, the current feature points, and the matching information; 41. The system of any one of items 35 to 40. [Matter 42] The control unit displaying on a display screen some or all of the image captured by an imaging device, the target image, the feature points, the target feature points, the current image, the current feature points, the matching information, the target feature point cluster, the target feature point cluster representative information, the current feature point cluster, and the current feature point cluster representative information; 37. The system of claim 36. [Matter 43] the control unit causes the display screen to display an instruction input unit related to an operation of the imaging device and / or an instruction input unit related to an operation to register information in a storage unit. 43. The system according to item 41 or 42. [Explanation of symbols]

[0164] 1···Robot / 2···End effector / 3···Image capture device / 4···Workpiece / 100···Controller / 101···Control device / 102···Input device / 103···Output device / 104···Storage device / 105···Image acquisition unit / 106···Feature point extraction unit / 107···Feature matching unit / 108···Robot control unit / 109···Feature point clustering unit / 110···Feature priority setting unit / 111···Feature complementation unit / F100···Target feature point / F101···Current feature point / F102···Target feature point for which a corresponding candidate feature point FP2 cannot be detected / F103···Feature Current feature points added by feature complementation processing / F104··Current feature points deleted by feature complementation processing / F110, F120, F130, F140··Target feature point cluster / F115, F125, F135, F145··Target feature point cluster representative information / F111, F121, F131, F141··Current feature point cluster / F116, F126, F136, F146··Current feature point cluster representative information / FP1··Feature point / FP2··Candidate feature point / V100··Image data display / V101··Shooting button / V102··Register button / V103··Parameter input field

Claims

1. A control method for controlling a device having a moving part by visual servoing, comprising: acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; generating a control signal for moving the movable part using matching information based on the matching between the current feature points and the target feature points obtained by the matching process; classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; the matching information includes the current feature point cluster representative information and the target feature point cluster representative information; A control method comprising:

2. The target feature point has a degree of freedom of information of ψ, the control signal moves the movable part with n degrees of freedom; The number N of the target feature points extracted from the target image satisfies N≧2×L, where L is the smallest integer M that satisfies M>(n+1) / ψ.

2. The control method according to claim 1.

3. A priority is set for each of the extracted target feature points; In the matching process, a current feature point that corresponds to the target feature point having a high priority is preferentially extracted.

3. The control method according to claim 1 or 2.

4. a priority is set for each of the classified target feature point clusters; In the matching process, a current feature point cluster associated with the target feature point cluster having a higher priority is preferentially acquired.

3. The control method according to claim 1 or 2.

5. A control method for controlling a device having a moving part by visual servoing, comprising: acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points from the target image by image processing; classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; generating a control signal for moving the movable part using matching information between the representative information of the current feature point cluster and the representative information of the target feature point cluster; A control method comprising:

6. A control method for controlling a device having a moving part by visual servoing, comprising: acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; correcting the current feature points using priorities set for the current feature points and the target feature points obtained by the matching process; generating a control signal for moving the movable part using matching information between the corrected current feature points and the target feature points; A control method comprising:

7. The target feature point has a degree of freedom of information of ψ, the control signal moves the movable part with n degrees of freedom; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; 7. The control method according to claim 5 or 6.

8. N≧1+(n+1) / ψ is satisfied.

7. The control method according to claim 5 or 6.

9. When L is the smallest integer M that satisfies M>(n+1) / ψ, N≧L+1 is satisfied.

7. The control method according to claim 5 or 6.

10. N≧2×L is satisfied.

7. The control method according to claim 5 or 6.

11. The number N of the target feature points extracted from the target image satisfies N≧3×L, where L is the smallest integer M that satisfies M>(n+1) / ψ.

7. The control method according to claim 1, 2, 5 or 6.

12. N≦L+40 is satisfied.

7. The control method according to claim 1, 2, 5 or 6.

13. N≦10×L is satisfied.

13. The control method according to claim 12.

14. n is 6 or 7 and ψ is 2 or 3; 14. The control method according to claim 13.

15. the number k of the plurality of target feature point clusters and the degree of freedom φ of information held by each target feature point cluster satisfy n+1≦φk≦ψN / 2, 3. The control method according to claim 1 or 2.

16. in the second clustering process, the current feature points associated with the target feature points in the matching process are classified into the current feature point clusters corresponding to the target feature point clusters to which the target feature points belong; 3. The control method according to claim 1 or 2.

17. In the first clustering process, the target feature points are classified into the plurality of target feature point clusters based on distances in the target image. and / or In the second clustering process, the current feature points are classified into the plurality of current feature point clusters based on distances in the current image.

3. The control method according to claim 1 or 2.

18. In the first clustering process, the target feature points are classified into the plurality of target feature point clusters based on gradation levels in the target image. and / or In the second clustering process, the current feature points are classified into the plurality of current feature point clusters based on gradations in the current image.

3. The control method according to claim 1 or 2.

19. In the first clustering process, the target feature points are classified into the plurality of target feature point clusters based on a shape model of the object and / or a shape model of the movable part. and / or In the second clustering process, the current feature points are classified into the plurality of current feature point clusters based on a shape model of the object and / or a shape model of the movable part.

3. The control method according to claim 1 or 2.

20. acquiring, as the target feature point cluster representative information, a center of gravity position of the target feature points included in each of the plurality of target feature point clusters; acquiring, as the current feature point cluster representative information, a center of gravity position of the current feature points included in each of the plurality of current feature point clusters; 3. The control method according to claim 1 or 2.

21. acquiring, as the target feature point cluster representative information, a center of gravity position and a magnification ratio of the target feature point included in each of the plurality of target feature point clusters, or the center of gravity position and a rotation angle, or the center of gravity position, the magnification ratio, and the rotation angle; acquiring, as the current feature point cluster representative information, a center of gravity position and a magnification ratio of the current feature point included in each of the plurality of current feature point clusters, or the center of gravity position and a rotation angle, or the center of gravity position, the magnification ratio, and the rotation angle; 3. The control method according to claim 1 or 2.

22. the priority order is set based on the detection frequency of the target feature points; 7. The control method according to claim 6.

23. the priority is set according to the contribution of the target feature point to visual servo control performance; 7. The control method according to claim 6.

24. The priority order is set based on the detection accuracy of the target feature points.

7. The control method according to claim 6.

25. The priority order is set using a group of images taken from a plurality of positions.

7. The control method according to claim 6.

26. the modified current feature point is generated using the current feature point with the lower priority.

7. The control method according to claim 6.

27. the modified current feature point is generated using another current feature point through a geometric transformation; 7. The control method according to claim 6.

28. The modified current feature points are generated using a physical model and a motion history.

7. The control method according to claim 6.

29. The image processing is processing using image features that are invariant to rotation, enlargement, and reduction.

7. The control method according to claim 1, 2, 5 or 6.

30. the target image and the current image are captured by an imaging device attached to the movable part; 7. The control method according to claim 1, 2, 5 or 6.

31. the target image and the current image are captured by an imaging device installed at a position where the movable part and the object can be imaged; 7. The control method according to claim 1, 2, 5 or 6.

32. The device having the movable part is a robot.

7. The control method according to claim 1, 2, 5 or 6.

33. A control program for causing a control unit to execute the control method according to any one of claims 1 to 6.

34. A computer-readable recording medium having the control program according to claim 33 recorded thereon.

35. The apparatus is an apparatus used for manufacturing an article, and the apparatus is controlled by the control method according to any one of claims 1, 2, 5, and 6 to manufacture the article. A method for manufacturing an article.

36. A system including a device having a movable part and a control unit, The control unit acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; generating a control signal for moving the movable part using matching information based on the matching between the current feature points and the target feature points obtained by the matching process; classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; the matching information includes the current feature point cluster representative information and the target feature point cluster representative information; A system characterized by:

37. The target feature point has a degree of freedom of information of ψ, the control signal moves the movable part with n degrees of freedom; The number N of the target feature points extracted from the target image satisfies N≧2×L, where L is the smallest integer M that satisfies M>(n+1) / ψ.

37. The system of claim 36.

38. A system including a device having a movable part and a control unit, The control unit acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points from the target image by image processing; classifying the target feature points into a plurality of target feature point clusters by a first clustering process; acquiring target feature point cluster representative information for each of the plurality of target feature point clusters; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; classifying the current feature points into a plurality of current feature point clusters by a second clustering process; acquiring current feature point cluster representative information for each of the plurality of current feature point clusters; generating a control signal for moving the movable part using matching information between the representative information of the current feature point cluster and the representative information of the target feature point cluster; A system characterized by:

39. A system including a device having a movable part and a control unit, The control unit acquiring a target image according to a relative positional relationship between the movable part and the target of the object; extracting target feature points from the target image by image processing; acquiring a current image according to a current relative positional relationship between the movable part and the object; extracting candidate feature points from the current image by image processing; performing a matching process between the candidate feature points and the target feature points, and extracting current feature points that correspond to the target feature points from among the candidate feature points; correcting the current feature points using priorities set for the current feature points and the target feature points obtained by the matching process; generating a control signal for moving the movable part using matching information between the corrected current feature points and the target feature points; A system characterized by:

40. The target feature point has a degree of freedom of information of ψ, the control signal moves the movable part with n degrees of freedom; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; 39. The system of claim 38.

41. The target feature point has a degree of freedom of information of ψ, the control signal moves the movable part with n degrees of freedom; the number N of target feature points extracted from the target image satisfies N>(n+1) / ψ; 40. The system of claim 39.

42. A priority is set for each of the extracted target feature points; In the matching process, a current feature point that corresponds to the target feature point having a high priority is preferentially extracted.

37. The system of claim 36.

43. a priority is set for each of the classified target feature point clusters; In the matching process, a current feature point cluster associated with the target feature point cluster having a higher priority is preferentially acquired.

39. The system of claim 38.

44. The device having a movable part is a robot.

44. A system according to any one of claims 36 to 43.

45. The control unit displaying a part or all of the image captured by the imaging device, the target image, the target feature points, the current image, the current feature points, and the matching information on a display screen; 37. The system of claim 36.

46. The control unit displaying on a display screen some or all of the image captured by an imaging device, the target image, the target feature points, the current image, the current feature points, the matching information, the target feature point cluster, the target feature point cluster representative information, the current feature point cluster, and the current feature point cluster representative information; 39. The system of claim 38.

47. the control unit causes the display screen to display an instruction input unit related to an operation of the imaging device and / or an instruction input unit related to an operation to register information in a storage unit.

47. The system of claim 45 or 46.

Images