Robot system, method for controlling a robot system, image processing device, image processing method, method for manufacturing an article, program, and recording medium

Abstract

To improve workability of a robot.SOLUTION: A robot device includes: a robot; an imaging unit; and a control unit which controls the robot. The control unit compares a predetermined image with a captured image of the robot captured by the imaging unit to acquire information regarding force and controls a force of the robot based on the information regarding the force.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to robot technology. [Background technology]

[0002] Visual servo technology is known for feeding back visual information acquired by visual sensors such as cameras to the motion control system of a robot. Visual servo consists of a fully closed-loop control system that controls the robot to minimize visual errors. Therefore, the robot can be positioned with high precision without external parameter calibration of the camera. Furthermore, because the robot is controlled based on visual information, it can flexibly respond to environmental changes such as variations in the position of target objects. For this reason, research and development aimed at improving the efficiency of tasks with large variations in the position of target objects has been accelerating in recent years.

[0003] Conventional vision servos control the position and speed of tools mounted on a robot based on visual errors. Therefore, for example, when performing contact-related tasks such as assembling parts using only vision servos, the robot will make contact with the object using position and speed control with high servo rigidity.

[0004] On the other hand, force control is a control method used to suppress excessive force applied to parts during contact-based operations. Force control is a control method that detects the force applied to a robot or tool and controls that force. In industrial robots, impedance control, which gives the tool a desired mechanical impedance characteristic, is often used as a form of force control. In such robotic force control operations, it is often necessary to position the tool or workpiece to a certain extent in the predetermined location where the operation will be performed. For example, in the assembly of a convex part and a concave part, the convex part is positioned up to the opening of the concave part.

[0005] Therefore, a method can be considered in which a visual servo is used to guide the robot to a predetermined position, and then the control is switched to force control from that position to have the robot perform the assembly. Patent Document 1 discloses a robot device that incorporates a force control system into a visual servo system and controls the robot based on the relative positional relationship between the assembly part and the part to be assembled. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2013-180380 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, when actually having a robot perform a task, the environment is not always ideal for the robot's work, for example, if there are obstacles within the robot's work area. Even in such situations, there is a need to improve the robot's work efficiency.

[0008] The present invention aims to improve the workability of robots. [Means for solving the problem]

[0009] A first aspect of this disclosure comprises a robot, an imaging unit, and a control unit for controlling the robot. It is a robotic system The control unit, Includes a first feature portion and a second feature portion The target image and the image captured by the imaging unit Includes a third feature portion corresponding to the object. Based on the current image, in the target image The First Feature section and the second feature portion and in the aforementioned current image The above 3 Information regarding the virtual mechanical properties acting between the feature portion and the robot is acquired, and the robot is controlled based on the information regarding the mechanical properties. This causes the robot to move the object. This is a robot system characterized by the following:

[0010] A second aspect of this disclosure is: A robot system comprising a robot, an imaging unit, and a control unit for controlling the robot, wherein the control unit acquires information regarding virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on a target image and a current image captured by the imaging unit, controls the robot based on the information regarding the mechanical properties, and displays at least one piece of information, such as the type of mechanical property acquired and the parameters of the mechanical property, on a display unit. That is the case.

[0011] A third aspect of this disclosure is: A robot system comprising a robot, an imaging unit, and a control unit for controlling the robot, wherein the control unit acquires information regarding virtual mechanical properties acting between feature portions in the target image and feature portions in the current image based on a target image and a current image captured by the imaging unit, controls the robot based on the information regarding mechanical properties, and the user can set whether or not to display the target image and the current image superimposed on the control unit. That is the case.

[0012] A fourth aspect of this disclosure is: A robot system comprising a robot, an imaging unit, and a control unit for controlling the robot, wherein the control unit acquires information regarding virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on a target image and a current image captured by the imaging unit, and controls the robot based on the information regarding the mechanical properties, wherein the target image includes a first feature portion corresponding to the object to be controlled and a second feature portion corresponding to an obstacle, and the control unit acquires a third feature portion corresponding to the object to be controlled from the current image, and acquires the mechanical properties acting between the third feature portion and the first feature portion, and the mechanical properties acting between the third feature portion and the second feature portion. A fifth aspect of the present disclosure is a robot system comprising a robot, an imaging unit, and a control unit for controlling the robot, wherein the control unit acquires information regarding virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on a target image and a current image captured by the imaging unit, controls the robot based on the information regarding mechanical properties, and displays a second user interface image on a display unit for accepting the setting of a feature portion in the target image, displays the target image in the second user interface image, and displays a user interface for acquiring a plurality of feature candidates that are candidates for feature portions from the target image. A sixth aspect of the present disclosure is a robot system comprising a robot, an imaging unit, and a control unit for controlling the robot, wherein the control unit acquires information regarding virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on a target image and a current image captured by the imaging unit, controls the robot based on the information regarding mechanical properties, and displays a second user interface image on a display unit that accepts the setting of a feature portion in the target image, and displays the target image and the current image superimposed on the second user interface image. A seventh aspect of this disclosure is a control method for a robot system, characterized in that, based on a target image including a first feature portion and a second feature portion and a current image including a third feature portion corresponding to an object captured by an imaging unit, information is obtained regarding virtual mechanical properties acting between the first and second feature portions in the target image and the third feature portion in the current image, and the robot is controlled based on the information regarding mechanical properties to move the object. That is the case. [Effects of the Invention]

[0013] According to the present invention, the workability of robots can be improved. [Brief explanation of the drawing]

[0014] [Figure 1] This is a schematic diagram of a robot device according to an embodiment. [Figure 2] This is a block diagram showing the control system of a robot device according to an embodiment. [Figure 3] This is a schematic diagram showing the working environment of the robotic device according to the embodiment. [Figure 4] (a) and (b) are schematic diagrams illustrating the assembly process according to the embodiment. [Figure 5] This is an explanatory diagram of a user interface (UI) image according to the embodiment. [Figure 6] This is a flowchart showing the process of setting configuration information according to the embodiment. [Figure 7] This is an explanatory diagram showing a target image according to the embodiment. [Figure 8] (a) and (b) are explanatory diagrams of UI images according to the embodiment. [Figure 9] (a) and (b) are explanatory diagrams of UI images according to the embodiment. [Figure 10] (a) and (b) are explanatory diagrams of UI images according to the embodiment. [Figure 11] (a) and (b) are explanatory diagrams of UI images according to the embodiment. [Figure 12](a) and (b) are explanatory diagrams of UI images according to the embodiment. [Figure 13] (a) and (b) are explanatory diagrams of UI images according to the embodiment. [Figure 14] This is a flowchart showing the force control used for assembly work according to the embodiment. [Figure 15] (a) and (b) are block diagrams of the control according to the embodiment. [Figure 16] (a), (b), and (c) are explanatory diagrams of the camera image display unit showing the assembly process according to the embodiment. [Figure 17] (a), (b), and (c) are explanatory diagrams of the camera image display unit showing the assembly process according to the embodiment. [Modes for carrying out the invention]

[0015] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. Figure 1 is a schematic diagram of a robot device 1000 according to an embodiment. The robot device 1000 comprises a robot 100, a servo control unit 230, a control device 400, an input device 500, a display 600, and a vision sensor 800.

[0016] The control device 400 is a device that controls the movement of the robot 100. The input device 500 is a device that allows the user to input various types of information. The display 600 is an example of a display unit and can display various images on the display screen 601. The vision sensor 800 is an example of an imaging unit and is, for example, a digital camera. The vision sensor 800 is a two-dimensional camera and can capture images of a subject and acquire two-dimensional image information. Note that the vision sensor 800 is not limited to a two-dimensional camera and may be, for example, a three-dimensional camera.

[0017] Robot 100 is, for example, an industrial robot and has a robot arm 200 and a robot hand 300. Robot 100 is installed on a manufacturing line and used to manufacture goods. The work of manufacturing goods includes, for example, having the robot hand 300 grasp a first workpiece and operating the robot arm 200 to assemble the first workpiece onto a second workpiece. The work of manufacturing goods also includes transport work, assembly work, processing work, and coating work. Processing work includes, for example, cutting work, grinding work, polishing work, and sealing work. The robot arm 200 is fitted with an end effector according to the work, and in this embodiment, the robot hand 300 is fitted to it.

[0018] In this embodiment, the robot arm 200 is a vertically articulated robot arm. The base end (fixed end) of the robot arm 200 is mounted on the base B1. A robot hand 300 is attached to the tip (free end), which is a predetermined location on the robot arm 200.

[0019] The robot arm 200 has a base 209, a plurality of links 210-216, and a plurality of joints J1-J6. The plurality of links 210-216 are connected in series in this order via the plurality of joints J1-J6. The robot arm 200 has joints J1, J2, J3, J4, J5, and J6, starting from the base end (link 210 side) and moving towards the tip end (link 216 side). Link 210, which is the base end of the robot arm 200, is fixed to the base 209. The base 209 is fixed to the upper surface of the base B1. Each link 211-216 is rotationally driven at each joint J1-J6. As a result, the robot arm 200 can adjust the robot hand 300 to any position and any orientation in the three axes.

[0020] The robot hand 300 is attached to a predetermined part of the robot arm 200, for example, the link 216 which is the tip. In other words, the link 216 is a support part configured to support an end effector such as the robot hand 300.

[0021] The posture of the robot arm 200 can be represented in a coordinate system. In Figure 1, coordinate system T0 is the coordinate system set on the base B1 to which the robot arm 200 is fixed. e This is the coordinate system set for the robot hand 300. Coordinate system T e This represents TCP (Tool Center Position). For example, coordinate system T e This is set in the robot hand 300. Coordinate system T0 and coordinate system T e It is represented by a three-axis orthogonal coordinate system consisting of the X, Y, and Z axes.

[0022] Coordinate system T c This is a coordinate system centered on the visual sensor 800, and coordinate system T o and coordinate system T e Similarly, it is represented by a three-axis orthogonal coordinate system consisting of the X, Y, and Z axes, and the optical axis direction of the vision sensor 800 is set to be the Z axis direction. In this embodiment, the vision sensor 800 is described as being fixed to a predetermined position relative to the coordinate system T0, for example, on the base B1, but it may also be fixed to the robot arm 200 or the robot hand 300.

[0023] The control device 400 is capable of controlling the movement, i.e., the posture, of the robot arm 200. The control device 400 is connected to a servo control unit 230, an input device 500, a display 600, and a vision sensor 800. If the input device 500 is, for example, a teaching pendant, it can be used by an operator (user) to teach the robot arm 200 how to move.

[0024] The servo control unit 230 drives and controls the motors 231 (Figure 2) of each joint J1 to J6. The servo control unit 230 is located, for example, inside the base 209. However, the location of the servo control unit 230 is not limited to inside the base 209 and may be located anywhere. For example, the servo control unit 230 may be located inside the housing of the control device 400. That is, the servo control unit 230 may be part of the configuration of the control device 400. In this embodiment, the control system 440 is configured including the input device 500, the display 600, the visual sensor 800, the control device 400, and the servo control unit 230. The control system 440 is also an example of an image processing device.

[0025] The servo control unit 230 drives and controls the motors 231 of each joint J1 to J6 so that the angle or torque of each joint J1 to J6 follows the command value obtained from the control device 400. In other words, the servo control unit 230 is configured to be able to control the position or torque of each joint of the robot 100.

[0026] Figure 2 is a block diagram showing the control system 440 of the robot device 1000 according to the embodiment. Each joint J1 to J6 has a motor 231, an angle sensor 250, and a torque sensor 260. Note that Figure 2 shows the configuration of one of the multiple joints J1 to J6 as a representative example.

[0027] The control device 400 is composed of a computer. The control device 400 has a CPU (Central Processing Unit) 401 as a processor.

[0028] The control device 400 also includes, as an example of a storage unit, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, and an HDD (Hard Disk Drive) 404. The control device 400 also includes a recording disk drive 405 and a plurality of input / output interfaces (I / F) 406 to 410.

[0029] The CPU 401 is connected to the ROM 402, RAM 403, HDD 404, recording disk drive 405, and interfaces 406-410 via the bus 420. The ROM 402 stores basic programs such as the BIOS. The RAM 403 is a storage device that temporarily stores various data, such as the results of calculations performed by the CPU 401.

[0030] The HDD 404 is a storage device that stores the results of calculations performed by the CPU 401 and various data acquired from external sources. The HDD 404 contains a program 430 that causes the CPU 401 to perform calculations. Based on the program 430 recorded (stored) in the HDD 404, the CPU 401 executes the image processing method, which is a control method described later, i.e., the manufacturing method of an item. The recording disk drive 405 can read various data and programs recorded on the recording disk 431.

[0031] An input device 500 is connected to interface 406. The CPU 401 acquires input data (input information) from the input device 500 via interface 406 and bus 420. A display 600 is connected to interface 407. Various images are displayed on the display 600 under the control of the CPU 401. Interface 408 is configured to allow connection of an external storage device 700, which is a storage unit such as rewritable non-volatile memory or an external HDD.

[0032] The servo control unit 230 is connected to the interface 409. The servo control unit 230 is connected to the motors 231, angle sensors 250, and torque sensors 260 of each joint J1 to J6 of the robot arm 200. The motors 231 are, for example, brushless DC motors or AC motors, and rotate-drive the corresponding joints via a reduction gear (not shown). The angle sensors 250 are, for example, rotary encoders, and are mounted on the motors 231 and configured to detect the rotation angle of the motors 231. The torque sensors 260 are mounted on the corresponding joints and configured to detect the torque acting on the corresponding joints.

[0033] The CPU 401 can acquire angle information from the angle sensor 250 and torque information from the torque sensor 260 via the servo control unit 230, interface 409, and bus 420. The servo control unit 230 may also divide the angle of the motor 231 detected using the angle sensor 250 by the reduction ratio of a reduction gear (not shown) to convert it into angle information for the corresponding joint and transmit it to the CPU 401.

[0034] The CPU 401 outputs command value data corresponding to each joint J1 to J6 to the servo control unit 230 via the bus 420 and interface 409 at predetermined time intervals (for example, 1 ms).

[0035] A vision sensor 800 is connected to interface 410. The vision sensor 800 takes images at predetermined time intervals (for example, 30 ms) under the control of the CPU 401. As a result, the CPU 401 can acquire visual information, i.e., captured image data, from the vision sensor 800 at predetermined time intervals (for example, 30 ms).

[0036] The HDD 404 is also a non-temporary recording medium that can be read by a computer. In this embodiment, the program 430 is stored in the HDD 404, but it is not limited to this. The program 430 may be recorded on any recording medium that is a non-temporary recording medium that can be read by a computer. For example, flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, and non-volatile memory can be used as recording media for supplying the program 430. Optical disks are, for example, disc media such as Blu-ray discs, DVDs, and CDs. Non-volatile memory is a storage device such as a USB memory stick, memory card, ROM, or SSD.

[0037] Next, the working environment in this embodiment will be described. Figure 3 is a schematic diagram showing the working environment of the robot device 1000 according to this embodiment. In this embodiment, the example will be described in which the predetermined task to be performed by the robot 100 is the task of assembling a workpiece W1, which is gripped by the robot hand 300, onto a workpiece W2 fixed to a workpiece fixing jig M1. The area A10 shown by the dashed line in Figure 3 is the area captured by the vision sensor 800. As shown in Figure 3, the robot 100 can hold the workpiece W1 by having the robot hand 300 grip the workpiece W1. Workpiece W1 is an example of an object to be held. Also, workpiece W1 is an example of a first workpiece, and workpiece W2 is an example of a second workpiece.

[0038] Figures 4(a) and 4(b) are schematic diagrams illustrating the assembly process according to the embodiment. Figure 4(a) shows the initial state of workpiece W1. Figure 4(b) shows the state after workpiece W1 has been assembled to workpiece W2. The assembly of workpiece W1 to workpiece W2 is performed by controlling the robot arm 200 so that workpiece W1 changes from the initial state shown in Figure 4(a) to the assembled state shown in Figure 4(b).

[0039] Here, when teaching the robot that the workpiece W1 is assembled, the robot hand 300 is operated to reach the assembled state shown in Figure 4(b). The robot hand 300 can be operated by the user by backdriving the robot arm 200, or by the user detaching the robot hand 300 from the robot arm 200 and operating it directly.

[0040] Figure 5 is an explanatory diagram of a user interface (UI) image 900 according to an embodiment. The UI image 900 is an example of a second user interface image and is an image that accepts settings for feature parts described later. The CPU 401 of the control device 400 displays the UI image 900 of Figure 5 on the display screen 601 of the display 600 of Figure 1 by executing the program 430. The UI image 900 can be operated by the user using the input device 500. The user can make various settings related to the operation of assembling workpiece W1 to workpiece W2 via the UI image 900.

[0041] UI image 900 includes a camera image display unit 901, a target image display checkbox 902, a current image display checkbox 903, a target image acquisition button 904, a point extraction button 905, a line extraction button 906, and a surface extraction button 907. UI image 900 also includes a save button 908, an execute button 909, a list display unit 950, a register button 951, a delete button 952, and a parameter display unit 960. That is, the CPU 401 displays the camera image display unit 901, the target image display checkbox 902, the current image display checkbox 903, the target image acquisition button 904, the point extraction button 905, the line extraction button 906, and the surface extraction button 907 in UI image 900. The CPU 401 also displays the save button 908, the execute button 909, the list display unit 950, the register button 951, the delete button 952, and the parameter display unit 960 in UI image 900.

[0042] Figure 6 is a flowchart showing the process of setting configuration information according to the embodiment. The CPU 401 executes the process according to the flowchart shown in Figure 6 by executing the program 430. Figure 7 is an explanatory diagram showing the target image according to the embodiment. Figures 8(a) to 13(b) are explanatory diagrams of the UI image according to the embodiment.

[0043] In step S101, when the user operates the target image acquisition button 904, the CPU 401 acquires a target image Ig (Figure 7), which is an example of a predetermined image, from the vision sensor 800. The target image Ig may also be acquired from the HDD 404, external storage device 700, or network. The target image Ig is temporarily stored in, for example, the RAM 403. The target image Ig is image data (visual information) that captures the assembled state shown in Figure 4(b). That is, the target image Ig includes the robot 100 and the workpiece W1 in a predetermined posture, as well as obstacles such as the workpiece fixing jig M1. In other words, the target image Ig is an image captured by the vision sensor 800 of the robot 100, workpiece W1, and workpiece W2 in a target posture where workpiece W1 is assembled onto workpiece W2 in a predetermined posture.

[0044] When the CPU 401 acquires the target image Ig, it turns on the target image display checkbox 902, as shown in Figure 8(a), and displays the target image Ig on the camera image display unit 901. The target image display checkbox 902 can also be operated by the user. The CPU 401 can switch the display / hide of the target image Ig on the camera image display unit 901 depending on the ON / OFF state of the target image display checkbox 902.

[0045] The point extraction button 905, the line extraction button 906, and the surface extraction button 907 are each examples of the first button. In step S102, when the point extraction button 905, the line extraction button 906, or the surface extraction button 907 is operated, the CPU 401 obtains feature quantities corresponding to the operated button from the target image Ig. Feature quantities in an image (image features) are, for example, feature points, feature lines, or feature surfaces.

[0046] Figure 8(b) illustrates the case when the user operates the point extraction button 905. In step S102, when the user operates the point extraction button 905, the CPU 401 extracts feature points P(n) = P(1) ~ P(N) from the target image Ig as features. P) is extracted. Feature points P(1) to P(N P ) are candidates for the feature part to be set (feature candidates).

[0047] Feature point P(n) is a point with a large change in luminance or color in the target image Ig, and can be extracted by, for example, the AKAZE algorithm. N P is the number of extracted feature points. Note that the feature point extraction algorithm is not limited to AKAZE, and may be, for example, SIFT, SURF, or ORB.

[0048] Then, as shown in FIG. 8(b), the CPU 401 superimposes and displays a figure indicating the feature points P(1) to P(N P ) on the target image Ig displayed on the camera image display unit 901. The figure indicating the feature points P(1) to P(N P ) is, for example, a quadrilateral.

[0049] In step S103, the CPU 401 waits for any one of the feature points P(1) to P(N P ) to be selected by the user. In the camera image display unit 901, the figure indicating the feature points P(1) to P(N P ) is a figure that can be selected by the user. When a figure of any feature point is selected by the user in the camera image display unit 901, the CPU 401, as shown in FIG. 9(a), highlights the selected figure and sets the feature point corresponding to the selected figure as the feature point to be set. That is, the user can select a feature point from among the plurality of feature points P(1) to P(N P ) via the UI image 900. The highlighting of the figure includes, for example, displaying the figure with a thick line, displaying the figure in a color different from the figures of other feature points, etc. In FIG. 9(a), the selected feature points are feature points Pc1 and Pc2.

[0050] In step S104, the CPU 401 waits for the user to operate the registration button 951. When the user operates the registration button 951, the CPU 401 displays the information of the selected feature points Pc1 and Pc2 on the list display unit 950 as shown in Figure 9(b), and temporarily stores it in the RAM 403. In this embodiment, registering information means temporarily storing the information in the RAM 403.

[0051] The registration information includes, for example, number information, name information, type information, controllability information, and feature descriptor information. The number is a number assigned in the order of registration. The name is a unique name assigned to the registered feature point. CPU401 automatically assigns names in the order of registration, for example, f1 to fnr (where nr is the registration number). For example, feature point Pc1 is assigned "f1", and feature point Pc 2 The character "f2" is assigned to this character. In the camera image display unit 901, "f1" is displayed near the character corresponding to feature point Pc1, and "f2" is displayed near the character corresponding to feature point Pc2. The names may be changed by the user.

[0052] The types are, for example, points, lines, and surfaces, and the user can select from a pull-down list. If the registration button 951 is operated while a feature point is selected as the feature quantity, the CPU 401 will automatically select "point" as the type. If the registration button 951 is operated while a feature line is selected as the feature quantity, the CPU 401 will automatically select "line" as the type. If the registration button 951 is operated while a feature surface is selected as the feature quantity, the CPU 401 will automatically select "surface" as the type.

[0053] The controllability value indicates whether the selected feature can be controlled together with the robot 100, and the user can select "Yes" or "No" from a pull-down list. "Yes" corresponds to controllable objects that can be moved by the robot 100, such as the robot 100 and the workpiece W1 held by the robot 100. "No" corresponds to obstacles that cannot be moved by the robot 100. Feature points Pc1 and Pc2 are feature quantities associated with the workpiece W1 grasped by the robot hand 300. Therefore, the user selects "Yes" for controllability.

[0054] The feature descriptor is an internal parameter used by the CPU 401 to obtain coordinates corresponding to registered features from images acquired from the vision sensor 800, and is not displayed on the list display unit 950.

[0055] Furthermore, the CPU 401 highlights the image representing the feature selected by the user from among the registered features in the UI image 900. For example, let's consider the case where the selected features are feature points Pc1 and Pc2. In the camera image display unit 901, the shape labeled "f1" and the shape labeled "f2" are highlighted with a thick line or the like. Also, in the list display unit 950, the entire row displaying the information for feature point Pc1 and the entire row displaying the information for feature point Pc2 are highlighted by being surrounded by a thick border or the like.

[0056] In step S105, it is determined whether the registration of the features necessary for the assembly work has been completed. If registration is not completed (S105: NO), the process in steps S102 to S104 is repeated. If registration is completed (S105: YES), the process moves on to the next step, S106.

[0057] The above describes the process of registering a controlled object. However, since it is necessary to avoid the controlled object coming into contact with an obstacle, the process of registering an obstacle will now be described. In this embodiment, the controlled object is either the workpiece W1 or the robot 100. In this embodiment, the obstacle is the workpiece fixing jig M1.

[0058] In step S102, when the user operates the line extraction button 906, the CPU 401 extracts the feature line L(n) = L(1) ~ L(N) from the target image Ig as a feature. L Extract the feature lines L(1)~L(N). L ) represents the candidate feature part to be defined (feature candidate).

[0059] Then, as shown in Figure 10(a), the CPU 401 generates feature lines L(1) to L(N) on the target image Ig displayed on the camera image display unit 901. L The figures representing ) are superimposed. Feature lines L(1)~L(N) L The shape representing the region is, for example, a rectangle representing the border. The border is, for example, a dashed line. The feature line L(n) is a line that indicates the boundary of the region and is extracted, for example, by the Hough transform. L This is the number of feature lines extracted.

[0060] In step S103, CPU401 generates feature lines L(1)~L(N L The camera image display unit 901 waits for the user to select one of the feature lines L(1) to L(N). L The shapes representing the features are shapes that the user can select. When the user selects any of the feature line shapes in the camera image display unit 901, the CPU 401 highlights the selected shape as shown in Figure 10(b) and sets the feature line corresponding to the selected shape as the feature line to be set. That is, the user selects multiple feature lines L(1) to L(N) via the UI image 900. L ) Feature lines can be selected from among them. Highlighting of shapes can be done, for example, by displaying the shape with a thick line, or by highlighting the shape with other features. line This includes displaying the shape in a different color. In Figure 10(b), the selected feature lines are feature lines Lc1 and Lc2.

[0061] In step S104, the CPU 401 waits for the user to operate the registration button 951. When the user operates the registration button 951, the CPU 401 displays the information of the selected feature lines Lc1 and Lc2 on the list display unit 950 as shown in Figure 11(a), and temporarily stores it in the RAM 403.

[0062] Here, in the camera image display unit 901, "f3" is displayed near the figure corresponding to feature line Lc1, and "f4" is displayed near the figure corresponding to feature line Lc2. Also, since the registration button 951 was operated with feature lines selected as the feature quantity, the CPU 401 automatically selects "line". Feature lines Lc1 and Lc2 are feature quantities associated with the workpiece fixing jig M1. For this reason, the user selects "no" as the controllability.

[0063] Furthermore, if the delete button 952 is operated while a registered feature is selected, the CPU 401 disables the highlighting of the selected feature and its display in the list display unit 950, and deletes the information temporarily stored in RAM 403.

[0064] If the registration of features necessary for the assembly work is completed in step S105, the process proceeds to step S106. Hereafter, the registered features Pc1, Pc2, Lc1, and Lc2 will also be referred to as features f1, f2, f3, and f4, corresponding to their names. Furthermore, when these features f1, f2, f3, and f4 are not distinguished, each feature f1, f2, f3, and f4 will also be referred to as feature f. In the target image Ig, features f1 and f2 are the first feature parts corresponding to the controlled object such as the workpiece W1, and features f3 and f4 in the target image Ig are the second feature parts corresponding to obstacles such as the workpiece fixing jig M1.

[0065] In step S106, the CPU 401 receives user input in the parameter display unit 960 for the mechanical characteristics (control parameters) of the virtual force acting on the feature quantity f registered in steps S102 to S105.

[0066] Figure 11(b) shows an example of force control parameters input to the parameter display unit 960. The force control parameters include, for example, number information, information on the name of the feature quantity associated with the mechanical properties, information on the type of mechanical property, and a numerical value for the type of mechanical property (the first parameter or second parameter, which will be described later). Figure 11(b) shows an example of input rows for six force control parameters numbered "1" to "6".

[0067] To bring workpiece W1 to the state of target image Ig, the user enters "f1" as the name of the feature associated with the mechanical properties in the input row of force control parameters number "1," and enters "IMP," which represents impedance, i.e., attractive force, as the type of mechanical property. Furthermore, to bring workpiece W1 to the state of target image Ig, the user enters "f2" as the name of the feature associated with the mechanical properties in the input row of force control parameters number "2," and enters "IMP," which represents impedance, i.e., attractive force, as the type of mechanical property.

[0068] There are two fields, "1" and "2," for entering the names of the features that link the mechanical properties. Each field, "1" and "2," can accept the names "f1" through "f4" mentioned above. Field "1" corresponds to the newly acquired image (image Ic, described later), while field "2" corresponds to the target image Ig. If the same name, for example "f1," is entered in both fields "1" and "2," the feature f1 registered in the target image Ig will be linked to the feature extracted from the newly acquired image from the visual sensor 800 that matches feature f1. Similarly, if the same name, for example "f2," is entered in both fields "1" and "2," the feature f2 registered in the target image Ig will be linked to the feature extracted from the newly acquired image from the visual sensor 800 that corresponds to feature f2.

[0069] Furthermore, in order for workpiece W1 to avoid the workpiece fixing jig M1, that is, for feature quantity f1 to receive a virtual repulsive force from feature quantities f3 and f4, information for linking mechanical properties is entered by the user in the input rows of force control parameters numbered "3" and "4". Similarly, in order for workpiece W1 to avoid the workpiece fixing jig M1, that is, for feature quantity f2 to receive a virtual repulsive force from feature quantities f3 and f4, information for linking mechanical properties is entered by the user in the input rows of force control parameters numbered "5" and "6".

[0070] Specifically, in the input row for force control parameter number "3", the user enters the name "f1" in the field "1" for entering the name of the feature quantity, and the name "f3" in the field "2" for entering the name of the feature quantity. In addition, the user enters "repulsion" as the type of mechanical property, indicating a repulsive force. Similarly, in the input row for force control parameter number "4", the user enters the name "f1" in the field "1" for entering the name of the feature quantity, and the name "f4" in the field "2" for entering the name of the feature quantity. In addition, the user enters "repulsion" as the type of mechanical property, indicating a repulsive force.

[0071] Furthermore, in the input row for force control parameter number "5", the user enters the name "f2" in the feature name input field "1" and the name "f3" in the feature name input field "2". Also, the user enters "repulsion" as the type of mechanical property, indicating repulsion. Furthermore, in the input row for force control parameter number "6", the user enters the name "f2" in the feature name input field "1" and the name "f4" in the feature name input field "2". Also, the user enters "repulsion" as the type of mechanical property, indicating repulsion.

[0072] Furthermore, for each input row of force control parameters numbered "1" to "6", the user inputs a value for the mechanical characteristic (first parameter or second parameter). When the user selects the corresponding location (input row) on the parameter display unit 960, the CPU 401 displays a UI image 970 (Figure 12(a)) corresponding to "IMP" on the display screen 601 of the display 600 if the type of mechanical characteristic of the selected location is "IMP". Also, if the type of mechanical characteristic of the selected location is "rebound", the CPU 401 displays a UI image 980 (Figure 12(b)) corresponding to "rebound" on the display screen 601 of the display 600. UI images 970 and 980 are examples of first user interface images.

[0073] For example, the force control parameters numbered "1" and "2" have a mechanical characteristic type of "IMP". Therefore, when the input row for force control parameter numbered "1" or "2" is selected in the parameter display unit 960, the CPU 401 displays the UI image 970 (Figure 12(a)) on the display screen 601 of the display 600.

[0074] UI image 970 displays a model diagram of mechanical impedance 971, a spring constant input section 972, a damper coefficient input section 973, an OK button 974, and a Cancel button 975. The spring constant (K) is input by the user into the spring constant input section 972. The damper coefficient (D) is input by the user into the damper coefficient input section 973.

[0075] When the user presses the OK button 974, the CPU 401 temporarily saves the values ​​entered in the spring coefficient input section 972 and the damper coefficient input section 973 to the RAM 403 and closes the UI image 970. When the Cancel button 975 is pressed, the CPU 401 closes the UI image 970 without saving the values ​​to the RAM 403. The spring coefficient (K) and damper coefficient (D) are examples of the first parameters. In summary, the CPU 401 accepts the settings for the spring coefficient (K) and damper coefficient (D) via the UI image 970.

[0076] For example, the force control parameters numbered "3" to "6" have a mechanical characteristic of the type "rebound". Therefore, when any input row of force control parameters numbered "3" to "6" is selected in the parameter display unit 960, the CPU 401 displays the UI image 980 (Figure 12(b)) on the display screen 601 of the display 600.

[0077] UI image 980 displays a graph 981 showing the relationship between distance and repulsion, a repulsion coefficient input section 982, an OK button 983, and a Cancel button 984. The repulsion coefficient (R) is entered by the user into the repulsion coefficient input section 982.

[0078] When the user presses the OK button 983, the CPU 401 temporarily saves the value entered in the repulsion coefficient input unit 982 to the RAM 403 and closes the UI image 980. When the Cancel button 984 is pressed, the CPU 401 closes the UI image 980 without saving the value to the RAM 403. The repulsion coefficient (R) is an example of a second parameter. In summary, the CPU 401 accepts the setting of the repulsion coefficient (R) via the UI image 980.

[0079] In step S106, the mechanical characteristics (force control parameters) of the virtual force are registered. The registered values ​​(first parameter and second parameter) are displayed in the "Value" column on the camera image display unit 901, as shown in Figure 13(a).

[0080] Furthermore, the CPU 401 superimposes a figure representing the mechanical properties of the registered virtual force onto the target image Ig in the camera image display unit 901. The mechanical properties of the virtual force in force control parameters numbered "3" to "6" are "repulsion," indicating a repulsive force, and therefore, as shown in Figure 13(a), repulsion regions a1 and a2 are displayed, for example, by shading. Repulsion regions a1 and a2 indicate the regions where the virtual repulsive force that generates feature quantities f3 and f4 is, for example, 10N.

[0081] In step S107, if the registration of the mechanical properties of the virtual force to the feature quantity f is completed (S107:YES), the CPU 401 proceeds to the processing of step S108; otherwise (S107:NO), the CPU 401 returns to the processing of step S106.

[0082] In step S108, when the user operates the save button 908, the CPU 401 saves the information including the registered feature quantity f and force control parameters to the HDD 404 as setting information PS. That is, setting information PS including the feature quantity f and force control parameters is set. This completes the setting flow. In this embodiment, setting information means saving the information to storage such as the HDD 404.

[0083] The user can verify whether the workpiece assembly settings are as intended using the UI image 900. As shown in Figure 13(b), when the user turns on the current image display checkbox 903, the CPU 401 displays the new captured image Ic acquired from the visual sensor 800 on the camera image display unit 901, superimposed on the target image Ig. For example, the target image Ig may be made semi-transparent and the target image Ig and captured image Ic may be superimposed and displayed on the camera image display unit 901.

[0084] Here, feature quantities f1 and f2 correspond to the workpiece W1, which is the object to be controlled, and feature quantities f3 and f4 correspond to the workpiece fixing jig M1, which is an obstacle. The CPU 401 obtains feature quantities that match feature quantities f1 and f2 in the newly acquired image Ic, for example, by pattern matching processing.

[0085] The CPU 401 displays the feature quantities f1 and f2 on the target image Ig as the character images "f1g" and "f2g" along with a rectangular shape on the camera image display unit 901. Hereafter, the feature quantities f1 and f2 on the target image Ig will be referred to with the same symbols as the character images "f1g" and "f2g", i.e., feature quantities f1g and f2g. As mentioned above, feature quantities f1g and f2g are examples of the first feature portion. The target image Ig and the feature quantities f1g, f2g, f3, f4, etc. are pre-configured information.

[0086] Similarly, the CPU 401 displays the feature quantities that match the feature quantities f1g and f2g on the captured image Ic as the character images "f1c" and "f2c" along with a rectangular shape in the camera image display unit 901. Hereafter, the feature quantities that match the feature quantities f1g and f2g on the captured image Ic will be referred to as f1c and f2c, with the same symbols as the character images "f1c" and "f2c". Feature quantities f1c and f2c are examples of the third feature portion.

[0087] Feature quantities f1g and f1c are associated by a virtual impedance characteristic (virtual attractive force) based on the setting information PS, and feature quantities f2g and f2c are associated by a virtual impedance characteristic (virtual attractive force) based on the setting information PS. That is, the CPU 401 sets (defines) the virtual attractive force acting between feature quantities f1g and f1c based on the setting information PS, and sets (defines) the virtual attractive force acting between feature quantities f2g and f2c based on the setting information PS. The CPU 401 displays the virtual attractive force between feature quantities f1g and f1c as a figure s1 on the camera image display unit 901. The CPU 401 also displays the virtual attractive force between feature quantities f2g and f2c as a figure s2 on the camera image display unit 901. Figures s1 and s2 are figures that the user can intuitively understand as representing attractive forces, such as triangular wave-shaped figures, i.e., spring figures. Figure s1 is a figure connecting the figure representing feature f1g and the figure representing feature f1c. Figure s2 is a figure connecting the figure representing feature f2g and the figure representing feature f2c. Hereafter, the virtual attractive force between feature f1g and feature f1c will have the same sign as figure s1, i.e., attractive force s1. Also, the virtual attractive force between feature f2g and feature f2c will be represented by figure s 2 Let the same sign be used for the attractive force s2. The attractive force s1 is obtained using the parameter set by force control parameter number "1", and the attractive force s2 is obtained using the parameter set by force control parameter number "2".

[0088] Furthermore, feature quantities f3 and f1c are associated with a virtual repulsive force based on the configuration information PS, and feature quantities f4 and f1c are associated with a virtual repulsive force based on the configuration information PS. Also, feature quantities f3 and f2c are associated with a virtual repulsive force based on the configuration information PS, and feature quantities f4 and f2c are associated with a virtual repulsive force based on the configuration information PS. In other words, CPU 401 sets (defines) the virtual repulsive force acting between feature quantities f3 and f1c based on the configuration information PS, and sets (defines) the virtual repulsive force acting between feature quantities f4 and f1c based on the configuration information PS. Similarly, CPU 401 sets (defines) the virtual repulsive force acting between feature quantities f3 and f2c based on the configuration information PS, and sets (defines) the virtual repulsive force acting between feature quantities f4 and f2c based on the configuration information PS. The calculation methods for these virtual forces (virtual attractive and repulsive forces) will be described later.

[0089] The user can instruct the control device 400 to start the assembly work from the UI image 900 or the input device 500. When using the UI image 900, the user can instruct the start of the work by operating the execute button 909. When using the input device 500, the user can instruct the start of the work by specifying the setting information PS on the input device 500.

[0090] Figure 14 is a flowchart showing force control for assembly work according to the embodiment. When the start of work is instructed, the CPU 401 and the servo control unit 230 execute processing according to the flowchart shown in Figure 14. Figures 15(a) and 15(b) are control block diagrams according to the embodiment. Here, the CPU 401 functions as the feedback control unit 450 shown in Figure 15(a) by executing the program 430. In this embodiment, the feedback control unit 450 and the servo control unit 230 are examples of control units (processing units), and the robot 100 can be force-controlled through their cooperation.

[0091] In this embodiment, a minor loop is configured in the servo control unit 230 to receive feedback of the torque detection value τ from the torque sensor 260. In addition, a major loop is configured in the feedback control unit 450 to receive feedback of the captured image Ic from the vision sensor 800 and the angle detection value q from the angle sensor 250. This major loop enables fully closed-loop control, which controls the robot 100 so that the error of the captured image Ic relative to the target image Ig is minimized.

[0092] The initial state of the workpiece W1 before force control begins is assumed to be the state shown in Figure 4(a). That is, until the state shown in Figure 4(a) is reached, the control device 400 and the servo control unit 230 control the robot 100 by position control. Position control is based on an angle command value and an angle detection value q. Position control brings the angle detection value q closer to the angle command value. When the state shown in Figure 4(a) is reached, the control device 400 and the servo control unit 230 begin force control of the robot 100.

[0093] In step S20, the feedback control unit 450 reads the target image Ig, setting information PS, and model information MO stored in the HDD 404. The model information MO includes information on link parameters used in kinematic and dynamic calculations of the robot arm 200, as well as information on the dynamic model.

[0094] In step S21, the feedback control unit 450 acquires the captured image Ic generated by the vision sensor 800 through imaging from the vision sensor 800.

[0095] Furthermore, in step S22, the feedback control unit 450 acquires the angle detection values ​​(joint angles) q of each joint J1 to J6 of the robot arm 200 via the servo control unit 230. The joint angle q is an angle value obtained by converting the angle acquired by the angle sensor 250 based on the reduction ratio of a reduction gear (not shown), but the conversion unit that performs this conversion is not shown in Figure 15(a). The function of this conversion unit may be provided by the control device 400 or by the servo control unit 230.

[0096] In step S23, the feedback control unit 450 controls the torque command value τ d The following is calculated. A block diagram of the processing in the feedback control unit 450 is shown in Figure 15(b).

[0097] As shown in Figure 15(b), the feedback control unit 450 includes a feature extraction unit 451, a virtual force calculation unit 452, a filter processing unit 453, a Jacobian calculation unit 454, a gravity-compensated torque calculation unit 455, and a torque command value calculation unit 456.

[0098] The feature extraction unit 451 extracts the feature quantities f set in the setting information PS from the target image Ig and the captured image Ic, respectively, based on the feature descriptors. For example, as shown in Figure 13(b), the feature extraction unit 451 extracts the feature quantities f1g, f2g, f3, and f4 from the target image Ig, and the feature quantities f1c and f2c from the captured image Ic. Note that since the feature quantities f1g, f2g, f3, and f4 are already set, the loaded information (data) may be used. As described above, the feature quantities f1g and f2g are examples of the first feature portion, the feature quantities f3 and f4 are examples of the second feature portion, and the feature quantities f1c and f2c are examples of the third feature portion.

[0099] The feature extraction unit 451 extracts the difference f of the feature quantities for which the dynamic relationship of a virtual force is set. e Calculate the vector (f is in bold). For example, the difference f of the feature corresponding to the force control parameter number "1". e1 The vector (f is in bold) is calculated according to the following equation (1).

number

[0100] Here, bold text indicates a vector or matrix. 1c The vector (f is in bold) represents the position of the feature quantities extracted from the captured image Ic on the image ([u 1c v 1c ] T ) is also f1g The vector (f is in bold) represents the position of the features extracted from the target image Ig on the image ([u 1g v 1g ] T ) The superscript "T" indicates the transpose of a matrix or vector.

[0101] The feature extraction unit 451 performs the same calculation for the difference of the feature corresponding to the other numbered force control parameters included in the setting information PS. In the example in Figure 13(b), there are six force control parameters, so the difference f e The vector (f is in bold) is a 12-dimensional vector, and f e =[f e1 T ... f e6 T ] T That is the case.

[0102] The feature extraction unit 451 then uses the difference f calculated in this manner. e The vector is passed to the virtual force calculation unit 452. Note that, in cases where the associated feature quantities are a combination of points and lines, such as the force control parameters numbered "3" to "6", the shortest distance between them is calculated as the difference between the feature quantities.

[0103] The virtual force calculation unit 452 calculates the feature difference f e Based on the force control parameters included in the setting information PS, a virtual force F acts between each feature quantity. v Calculate (F is in bold).

[0104] For example, the virtual force F corresponding to the force control parameter number "1" for which the virtual impedance characteristics are set. v1 The vector (F is in bold) is calculated according to equation (2) below.

number

[0105] In formula (2), K d1 ,D d1These are scalar values ​​representing the spring constant and damper constant in the virtual impedance characteristics corresponding to the force control parameter number "1," respectively.

[0106] Furthermore, a virtual force F corresponding to force control parameter number "3" with set virtual repulsive force characteristics is also defined. v3 The vector (F is in bold) is calculated according to equation (3) below.

number

[0107] In equation (3), R3 is a coefficient that relates distance and repulsion in the virtual repulsive force corresponding to the force control parameter number "3".

[0108] The feature extraction unit 451 performs similar calculations for virtual forces corresponding to other numbered force control parameters included in the setting information PS. In the example in Figure 13(b), there are six force control parameters, so the virtual force F v The vector (F is in bold) is a 12-dimensional vector, and F v =[F v1 T ... F v6 T ] T The feature extraction unit 451 then extracts the virtual force F calculated in this manner. v The vector is passed to the filter processing unit 453.

[0109] In this way, the feature extraction unit 451 compares the target image Ig and the captured image Ic to obtain information about force, specifically the virtual force F. v The vector is obtained. That is, the feature extraction unit 451 obtains a virtual attraction F based on the relationship between feature f1c and feature f1g. v1 , a virtual attraction F based on the relationship between feature f2c and feature f2g v2 The feature extraction unit 451 calculates the virtual repulsive force F based on the relationship between feature f1c and feature f3. v3 , a virtual repulsive force F based on the relationship between feature f1c and feature f4 v4 , a virtual repulsive force F based on the relationship between feature f2c and feature f3v5 , a virtual repulsive force F based on the relationship between feature f2c and feature f4 v6 We seek.

[0110] The filter processing unit 453 processes the virtual force F v A predetermined filtering process is performed on each element of the vector. The filter used for filtering is a digital filter obtained by discretizing the transfer function shown in equation (4) below using a bilinear transform or the like.

number

[0111] This transfer function is a second-order notch filter, which has the effect of reducing the gain in a specific frequency range and is used as a method to stabilize control. Here, s is the differential operator, ω n is the center frequency of the notch, ζ is the width of the notch, and d is the variable that determines the depth of the notch. Filtered virtual force F v This is then passed to the torque command value calculation unit 456.

[0112] The Jacobian calculation unit 454 calculates the image Jacobian J for each feature f. img (J is in bold) matrix and robotic Jacobian J r Calculate the matrix (J is in bold). Image Jacobian J img This is the coordinate system T set for the robot hand 300. e This is an 8x6 matrix that correlates the small displacement of the robot with the small displacement of the feature f. (Robot Jacobian J) r This refers to the minute displacement amounts of each joint J1 to J6 of the robot arm 200 and the coordinate system T set for the robot hand 300. e This is a 6x6 matrix that correlates the small displacement amounts with the image Jacobian J. img and the robot Jacobian J r Define.

number

[0113] Here, xe (x in bold) is the 6 - degree - of - freedom position vector x of coordinate system T in coordinate system To e of coordinate system T e =[X e Y e Z e α e β e γ e T where q (q in bold) is the joint - angle vector q = [q1 … q6] of each joint J1 - J6 of the robot arm 200 T is as follows

[0114] The Jacobian calculation unit 454 calculates an eight - row six - column integrated Jacobian J (J in bold) obtained by taking the inner product of the image Jacobian J img and the robot Jacobian J r and integrating them. That is, the integrated Jacobian J is a matrix that associates the motion of the robot arm 200 with the motion of the feature quantity

[0115] The gravity - compensation torque calculation unit 455 calculates a gravity - compensation torque τ (τ in bold) that balances the estimated value of the gravity torque generated by gravity at each joint J1 - J6 of the robot arm 200 based on the model information MO and the joint angles q of each joint J1 - J6 of the robot arm 200 g (τ in bold) is calculated. The gravity - compensation torque τ g is calculated, for example, by a method based on the derivation of the equation of motion by the Newton - Euler method

[0116] The torque command - value calculation unit 456 calculates a torque command value τ (τ in bold) according to the following formula (6) based on the filtered virtual force F v , the integrated Jacobian J, and the gravity - compensation torque τ g (τ in bold) is calculated d (τ in bold) is calculated [[ID=SO]]

Equation

[0117] In this way, the torque command - value calculation unit 456 obtains a torque command value τ (τ in bold) for force - controlling the robot 100 based on the virtual force F v and obtains a torque command value τ (τ in bold) for force - controlling the robot 100 d is obtained

[0118] In step S24, the feedback control unit 450 determines whether each element included in the feature difference f e is less than or equal to a predetermined threshold. If all elements are less than or equal to the predetermined threshold (S24: YES), the feedback control unit 450 and the servo control unit 230 end the process. Otherwise (S24: NO), the feedback control unit 450 proceeds to the process of step S25. The predetermined threshold is set to 3 pixels for the force control parameters numbered "1" and "2", for example. Note that pixel represents the unit per pixel of the image.

[0119] In step S25, the feedback control unit 450 acquires a torque detection value τ from the torque sensors 260 of the respective joints J1 to J6 of the robot arm 200.

[0120] In step S26, the servo control unit 230 calculates a current command value of the current supplied to the motors 231 of the respective joints J1 to J6 of the robot arm 200 so that the torque detection value τ follows the torque command value τ d . Then, the servo control unit 230 drives the motor 231 by supplying the current i d to the motor 231 based on the current command value. That is, the servo control unit 230 performs torque control based on the difference between the torque value τ and the torque command value τ d .

[0121] When the process of step S26 ends, the feedback control unit 450 returns to the process of step S21. Thus, every time a new captured image Ic is acquired by the feedback control unit 450, the processes of steps S21 to S26 are repeatedly executed. Through the above control processes, the control device 400 and the servo control unit 230 can control the robot arm 200 based on the virtual dynamic characteristics set by the user in the feature space.

[0122] Next, the process of assembling workpiece W1 to workpiece W2 using the configuration information PS will be explained in detail. Figures 16(a) to 17(c) are explanatory diagrams of the camera image display unit 901 showing the assembly work according to the embodiment.

[0123] First, when workpiece W1 is in its initial state, the positional relationship between workpiece W1, workpiece W2, workpiece fixing jig M1, and the associated feature quantity f is as shown in Figure 16(a).

[0124] When the control device 400 starts controlling the robot arm 200, the virtual impedance characteristics corresponding to the force control parameters numbered "1" and "2" attract feature quantities f1c and f1g, and feature quantities f2c and f2g. As a result, as shown in Figure 16(b), feature quantity f1c approaches feature quantity f1g, and feature quantity f2c approaches feature quantity f2g.

[0125] As shown in Figure 16(c), when feature f1c approaches the repulsion region a1, the workpiece W1 behaves as if it were touching an invisible wall before contacting the workpiece fixing jig M1, due to the action of the virtual repulsive force characteristics corresponding to the force control parameter number "3". At this time, the attractive motion between workpiece W1 and workpiece W2 continues due to the action of the virtual impedance characteristics corresponding to the force control parameters number "1" and "2". Therefore, as shown in Figure 17(a), workpiece W1 slides along the outer frame of the repulsion region a1 so that feature f1c approaches feature f1g and feature f2c approaches feature f2g. Thus, collision between workpiece W1 and the workpiece fixing jig M1 can be avoided.

[0126] As shown in Figure 17(b), when the feature f1c leaves the repulsion region a1, the workpiece W1 moves to coordinate system T o It begins to move in the X-axis direction. Then, as shown in Figure 17(c), the tip of workpiece W1 comes into contact with the opening of workpiece W2.

[0127] Then, due to the action of virtual impedance characteristics corresponding to force control parameters numbered "1" and "2", workpiece W1 conforms to the shape of workpiece W2, and the difference f of feature quantities e1 and f e2 When all of the components become 3 pixels or less, the control device 400 terminates the assembly process. This results in the production of an item in which workpiece W1 is assembled onto workpiece W2.

[0128] As described above, the feedback control unit 450 and the servo control unit 230 perform force control based on the difference between the target image Ig and the captured image Ic, thereby bringing the posture of the robot 100 closer to the target posture at the time the target image Ig was captured. That is, the position and posture of the workpiece W1 approach the position and posture in the target image Ig. When the posture of the robot 100 is approximately the target posture, the assembly of the workpiece W1 is completed.

[0129] As described above, according to this embodiment, the robot 100 is controlled so that a virtual force acts between the feature quantities f1c and f2c extracted from the captured image Ic and the set feature quantities f1g, f2g, f3, and f4. Therefore, force-based operations can be set in the feature quantity space, and the workability of the robot 100 can be improved.

[0130] For example, virtual attractive forces can be set for feature quantities f1g and f2g associated with the workpiece W1, which is the object to be controlled, and virtual repulsive forces can be set for feature quantities f3 and f4 associated with the workpiece fixing jig M1, which is an obstacle. Therefore, it becomes possible to perform both contact-based tasks such as assembly work and other actions such as obstacle avoidance. This reduces the likelihood of the workpiece W1 getting caught on an obstacle, such as the workpiece fixing jig M1, and reduces the frequency of assembly failures, thereby improving the workability of the robot 100.

[0131] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0132] In the above-described embodiment, the case in which the feedback control unit 450 is part of the functions of the CPU 401 and the servo control unit 230 is configured as a different device from the CPU 401 was explained, but the invention is not limited to this. The CPU 401 may be configured to implement some or all of the functions of the servo control unit 230 based on the program 430.

[0133] Furthermore, in the above-described embodiment, the control device 400 and the display 600 are used to display the UI image 900 and to accept user input, but the system is not limited to this. For example, an electronic device equipped with a CPU and a display device such as a display may be used separately. The electronic device may be an information processing device such as a desktop PC (Personal Computer), laptop PC, tablet PC, or smartphone. Also, if the input device 500 is a teaching pendant with a display device, the UI image may be displayed on that display device.

[0134] Furthermore, although the above-described embodiment described the case where the robot arm 200 is a vertically articulated robot arm, it is not limited to this. The robot arm 200 may be various types of robot arms, such as a horizontally articulated robot arm, a parallel link robot arm, or a Cartesian robot.

[0135] Furthermore, although the above-described embodiment described a case in which a robot hand 300 is attached to the robot arm 200, the invention is not limited to this. The robot arm 200 may be equipped with a holding mechanism capable of holding objects such as workpieces as an end effector. Examples of holding mechanisms include mechanisms that hold workpieces by suction. In addition, the robot arm 200 may be equipped with a tool for processing workpieces as an end effector.

[0136] Furthermore, although a robot was used in the above-described embodiment, the present invention is not limited thereto. For example, the present invention can also be applied to a machine that can automatically perform actions such as extending and retracting, bending and straightening, moving up and down, moving left and right, or rotating, or a combination thereof, based on information stored in a memory device provided in a control device.

[0137] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0138] The above disclosures of embodiments include the following configurations and methods.

[0139] (Composition 1) Robots and, Imaging unit, The robot comprises a control unit for controlling the robot, The control unit compares a predetermined image with an image captured by the imaging unit of the robot to obtain force information, and controls the robot based on the force information. A robotic device characterized by the following features.

[0140] (Configuration 2) The aforementioned predetermined image is an image of the robot in a predetermined posture, The control unit, by force control, brings the posture of the robot closer to the predetermined posture. The robot device according to configuration 1, characterized by the above.

[0141] (Composition 3) The aforementioned predetermined image includes a first feature portion corresponding to the object to be controlled and a second feature portion corresponding to the obstacle. The control unit, A third feature portion corresponding to the object to be controlled is obtained from the captured image. Based on the relationship between the third feature portion and the first feature portion, and the relationship between the third feature portion and the second feature portion, information regarding the force is obtained. A robotic device according to configuration 1 or 2, characterized by the above.

[0142] (Composition 4) The object to be controlled is the robot, or an object held by the robot. The robotic device according to configuration 3, characterized by the features described above.

[0143] (Composition 5) The control unit sets a virtual attractive force between the third feature portion and the first feature portion, and sets a virtual repulsive force between the third feature portion and the second feature portion, as information relating to the force. A robotic device according to configuration 3 or 4, characterized by the above.

[0144] (Composition 6) The control unit sets the virtual attractive force using a first parameter and sets the virtual repulsive force using a second parameter. The robotic device according to configuration 5, characterized by the features described herein.

[0145] (Composition 7) The control unit displays a first user interface image on the display unit that accepts the settings of the first parameter and the second parameter. The robot device according to configuration 6, characterized by the features described above.

[0146] (Composition 8) The control unit displays a second user interface image on the display unit that accepts the settings for the first feature portion and the second feature portion. A robotic device according to any one of configurations 3 to 7, characterized by the above.

[0147] (Composition 9) The control unit displays the predetermined image in the second user interface image. The robotic device according to configuration 8, characterized by the above.

[0148] (Composition 10) The control unit displays buttons in the second user interface image for obtaining a plurality of feature candidates that are candidates for the first feature portion and the second feature portion from the predetermined image. The robotic device according to configuration 9, characterized by the features described herein.

[0149] (Composition 11) The control unit, When the aforementioned button is operated, the first and second feature portions to be set are displayed on the predetermined image shown on the second user interface image, with the user able to select from the plurality of feature candidates a corresponding graphic for each of the plurality of feature candidates. A robotic device according to configuration 10, characterized by the above.

[0150] (Composition 12) The control unit, On the predetermined image displayed in the second user interface image, shapes corresponding to the set first feature portion and the second feature portion are displayed. A robotic device according to any one of the configurations 9 to 11, characterized by the features described herein.

[0151] (Composition 13) The control unit displays the predetermined image and the captured image superimposed in the second user interface image. A robotic device according to any one of the configurations 8 to 12, characterized by the features described above.

[0152] (Method 14) A method for controlling a robotic device, The system compares a predetermined image with an image captured by the imaging unit of the robot to obtain force information. The robot is force-controlled based on the force information. A control method for a robotic device characterized by the following:

[0153] (Composition 15) An image processing device that acquires force-related information for force control of a robot, The system includes a control unit that compares a predetermined image with an image captured by the imaging unit of the robot to acquire information about the force, An image processing apparatus characterized by the following:

[0154] (Method 16) An image processing method for acquiring force-related information for controlling the force of a robot, The predetermined image is compared with the image captured by the imaging unit, To obtain information regarding the aforementioned force, An image processing method characterized by the following:

[0155] (Method 17) A method for manufacturing articles, characterized by manufacturing articles using a robotic device according to any one of the configurations 1 to 13.

[0156] (Composition 18) A program for causing a computer to execute the control method of the robotic device described in Method 14.

[0157] (Composition 19) A computer-readable recording medium on which the program described in Configuration 18 is recorded. [Explanation of Symbols]

[0158] Ic... Captured image, Ig... Target image (predetermined image), 100... Robot, 230... Servo control unit (control unit), 450... Feedback control unit (control unit), 800... Vision sensor (imaging unit), 1000... Robot device

Claims

1. Robots and, Imaging unit, A robot system comprising a control unit for controlling the robot, The control unit, Based on a target image including a first feature portion and a second feature portion and a current image including a third feature portion corresponding to the object captured by the imaging unit, information regarding the virtual mechanical properties acting between the first and second feature portions in the target image and the third feature portion in the current image is obtained. The robot is controlled to move the object based on the information regarding the mechanical properties. A robotic system characterized by the following features.

2. A virtual attractive force is set for the first feature portion of the target image, and a virtual repulsive force is set for the second feature portion of the target image. Using the imaging unit, the current image is acquired in which the third feature portion corresponding to the first feature portion of the target image is captured. When the control unit brings the object corresponding to the third feature portion of the current image closer to the first portion corresponding to the first feature portion in the target image where the virtual attractive force is set, it controls the robot so that the object avoids the second portion corresponding to the second feature portion in the target image where the virtual repulsive force is set. The robot system according to feature 1.

3. A robot and Imaging unit, A robot system comprising a control unit for controlling the robot, The control unit, Based on the target image and the current image captured by the imaging unit, information regarding the virtual dynamic properties acting between the feature portion in the target image and the feature portion in the current image is obtained. The robot is controlled based on the information regarding the mechanical properties. The control unit displays at least one piece of information on the display unit, which includes the type of mechanical characteristic to be acquired and the parameters of the mechanical characteristic. A robotic system characterized by the following features.

4. A robot and Imaging unit, A robot system comprising a control unit for controlling the robot, The control unit, Based on the target image and the current image captured by the imaging unit, information regarding the virtual dynamic properties acting between the feature portion in the target image and the feature portion in the current image is obtained. The robot is controlled based on the information regarding the mechanical properties. The control unit allows the user to set whether or not to display the target image and the current image superimposed. A robotic system characterized by the following features.

5. The aforementioned target image is an image of the robot in a predetermined posture. The control unit controls the robot so that its posture approaches the predetermined posture. The robot system according to feature 1.

6. A robot and Imaging unit, A robot system comprising a control unit for controlling the robot, The control unit, Based on the target image and the current image captured by the imaging unit, information regarding the virtual dynamic properties acting between the feature portion in the target image and the feature portion in the current image is obtained. The robot is controlled based on the information regarding the mechanical properties. The aforementioned target image includes a first feature portion corresponding to the object to be controlled and a second feature portion corresponding to the obstacle. The control unit, From the current image, a third feature portion corresponding to the object to be controlled is obtained, The mechanical properties acting between the third feature portion and the first feature portion, and the mechanical properties acting between the third feature portion and the second feature portion are obtained. A robotic system characterized by the following features.

7. The object to be controlled is the robot, or an object held by the robot. The robot system according to feature 6.

8. The control unit acquires a virtual attractive force as the mechanical property acting between the third feature portion and the first feature portion, and acquires a virtual repulsive force as the mechanical property acting between the third feature portion and the second feature portion. The robot system according to feature 6.

9. The control unit displays a virtual attractive force acting between the third feature portion and the first feature portion on the target image or the current image using a first figure, and displays a virtual repulsive force acting between the third feature portion and the second feature portion using a second figure different from the first figure. The robot system according to feature 6.

10. The control unit acquires the virtual attractive force using the first parameter and the virtual repulsive force using the second parameter. The robot system according to claim 2, characterized in that it is the same as described in claim 2.

11. The control unit displays a first user interface image on the display unit that accepts the settings of the first parameter and the second parameter. The robot system according to claim 10.

12. The control unit displays a second user interface image on the display unit that accepts the setting of feature portions in the target image. The robot system according to feature 1.

13. The control unit displays the target image in the second user interface image. The robot system according to claim 12, characterized in that it is the robot system according to claim 12.

14. A robot and Imaging unit, A robot system comprising a control unit for controlling the robot, The control unit, Based on the target image and the current image captured by the imaging unit, information regarding the virtual dynamic properties acting between the feature portion in the target image and the feature portion in the current image is obtained. The robot is controlled based on the information regarding the mechanical properties. The control unit displays a second user interface image on the display unit that accepts the setting of feature portions in the target image, and displays a user interface in the second user interface image that displays the target image and obtains a plurality of feature candidates that are candidates for feature portions from the target image. A robotic system characterized by the following features.

15. The control unit, When the user interface is operated, the user can select the feature portion to be set from among the multiple feature candidates, and a shape corresponding to each of the multiple feature candidates is displayed on the target image shown in the second user interface image. The robot system according to feature 14.

16. The control unit, On the target image displayed in the second user interface image, shapes corresponding to each of the set feature parts are displayed. The robot system according to claim 12, characterized in that it is the robot system according to claim 12.

17. A robot and Imaging unit, A robot system comprising a control unit for controlling the robot, The control unit, Based on the target image and the current image captured by the imaging unit, information regarding the virtual dynamic properties acting between the feature portion in the target image and the feature portion in the current image is obtained. The robot is controlled based on the information regarding the mechanical properties. The control unit displays a second user interface image on the display unit that accepts the setting of feature portions in the target image, and displays the target image and the current image superimposed on the second user interface image. A robotic system characterized by the following features.

18. A method for controlling a robot system, Based on a target image including a first feature portion and a second feature portion and a current image including a third feature portion corresponding to the object captured by the imaging unit, information regarding the virtual mechanical properties acting between the first and second feature portions in the target image and the third feature portion in the current image is obtained. The robot is controlled to move the object based on the information regarding the mechanical properties. A control method characterized by the following:

19. A method for manufacturing an article, characterized by manufacturing the article using the robot system described in any one of claims 1 to 17.

20. A program for causing a computer to execute the control method described in claim 18.

21. A computer-readable recording medium having the program described in claim 20 recorded on it.

Images