Robot system, robot system control method, image processing device, image processing method, article manufacturing method, program, and recording medium

Abstract

To improve workability of a robot.SOLUTION: A robot device includes: a robot; an imaging unit; and a control unit which controls the robot. The control unit compares a predetermined image with a captured image of the robot captured by the imaging unit to acquire information regarding force and controls a force of the robot based on the information regarding the force.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to robot technology. [Background technology]

[0002] Visual servoing technology is known, which feeds back visual information acquired by visual sensors such as cameras to a robot's motion control system. Visual servoing is composed of a fully closed-loop control system that controls the robot to minimize visual errors. This allows the robot to be positioned with high precision without externally calibrating the camera's parameters. Furthermore, because the robot is controlled based on visual information, it can flexibly respond to environmental changes such as variations in the position of the target object. For this reason, research and development aimed at improving the efficiency of tasks involving large variations in the position of the target object has been accelerating in recent years.

[0003] Conventional visual servoing controls the position and speed of a tool attached to a robot based on visual error. Therefore, when performing contact tasks such as assembling parts using only visual servoing, the robot must bring the part into contact with the target object using position and speed control with high servo stiffness.

[0004] Meanwhile, force control is a method of controlling excessive force being applied to parts in contact-related tasks. Force control is a control method that detects the force applied to a robot or tool and controls that force. In industrial robots, impedance control, which gives the tool desired mechanical impedance characteristics, is often used as force control. In such robotic force control tasks, it is often necessary to position the tool or workpiece to a certain extent at a predetermined position where the task will be performed. For example, when assembling a convex part and a concave part, the convex part must be positioned up to the opening of the concave part.

[0005] Therefore, one possible method is to have the robot approach a predetermined position using visual servoing, and then switch to force control from the predetermined position to have the robot perform the assembly.Patent Document 1 discloses a robot device in which a force control system is incorporated into the visual servoing system, and the robot is controlled based on the relative positional relationship between the part to be assembled and the part to be assembled. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-180380 Summary of the Invention [Problem to be solved by the invention]

[0007] However, when a robot is actually tasked with performing a task, the environment is not always conducive to the robot's work, for example, there may be obstacles in the work area where the robot is to perform the task, etc. Even in such situations, there is a demand for improving the robot's workability.

[0008] An object of the present invention is to improve the workability of a robot. [Means for solving the problem]

[0009] A first aspect of the present invention is a robot device comprising a robot, an imaging unit, and a control unit that controls the robot, wherein the control unit compares a predetermined image with an image of the robot captured by the imaging unit to obtain information related to force, and performs force control on the robot based on the information related to force.

[0010] A second aspect of the present invention is a method for controlling a robot device, characterized in that a predetermined image is compared with an image of the robot captured by an imaging unit to obtain information regarding force, and the robot is force-controlled based on the information regarding force.

[0011] A third aspect of the present invention is an image processing device that acquires information about forces for force control of a robot, characterized in that it includes a control unit that acquires information about the forces by comparing a predetermined image with an image of the robot captured by an imaging unit.

[0012] A fourth aspect of the present invention is an image processing method for acquiring information regarding forces for force control of a robot, characterized in that a predetermined image is compared with an image of the robot captured by an imaging unit, and the information regarding the forces is acquired. [Effects of the Invention]

[0013] According to the present invention, the workability of a robot can be improved. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of a robot device according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing a control system of a robot device according to an embodiment. [Figure 3] FIG. 1 is a schematic diagram illustrating a working environment for a robot device according to an embodiment. [Figure 4] 5A and 5B are schematic diagrams for explaining an assembly operation according to the embodiment. [Figure 5] FIG. 2 is an explanatory diagram of a user interface (UI) image according to the embodiment. [Figure 6] 10 is a flowchart illustrating a process for setting setting information according to the embodiment. [Figure 7] FIG. 2 is an explanatory diagram showing a target image according to the embodiment; [Figure 8] 10(a) and 10(b) are explanatory diagrams of a UI image according to the embodiment. [Figure 9] 10(a) and 10(b) are explanatory diagrams of a UI image according to the embodiment. [Figure 10] 10(a) and 10(b) are explanatory diagrams of a UI image according to the embodiment. [Figure 11]10(a) and 10(b) are explanatory diagrams of a UI image according to the embodiment. [Figure 12] 10(a) and 10(b) are explanatory diagrams of a UI image according to the embodiment. [Figure 13] 10(a) and 10(b) are explanatory diagrams of a UI image according to the embodiment. [Figure 14] 10 is a flowchart illustrating force control for performing an assembly operation according to the embodiment. [Figure 15] 4A and 4B are block diagrams of control according to the embodiment. [Figure 16] 10A, 10B, and 10C are explanatory diagrams of a camera image display unit that displays the state of an assembly operation according to the embodiment. [Figure 17] 10A, 10B, and 10C are explanatory diagrams of a camera image display unit that displays the state of an assembly operation according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Fig. 1 is a schematic diagram of a robot device 1000 according to an embodiment. The robot device 1000 includes a robot 100, a servo control unit 230, a control device 400, an input device 500, a display 600, and a visual sensor 800.

[0016] The control device 400 is a device that controls the operation of the robot 100. The input device 500 is a device that can be operated by a user to input various types of information. The display 600 is an example of a display unit, and can display various types of images on a display screen 601. The visual sensor 800 is an example of an imaging unit, and is, for example, a digital camera. The visual sensor 800 is a two-dimensional camera, and can capture an image of a subject to acquire two-dimensional image information. Note that the visual sensor 800 is not limited to a two-dimensional camera, and may be, for example, a three-dimensional camera.

[0017] The robot 100 is, for example, an industrial robot, and includes a robot arm 200 and a robot hand 300. The robot 100 is installed on a production line and used to manufacture articles. The work of manufacturing the article includes, for example, an operation of gripping a first workpiece with the robot hand 300 and operating the robot arm 200 to assemble the first workpiece to a second workpiece. The work of manufacturing the article also includes a transport operation, an assembly operation, a processing operation, and a coating operation. The processing operation includes, for example, a cutting operation, a grinding operation, a polishing operation, and a sealing operation. An end effector appropriate for the operation is attached to the robot arm 200, and in this embodiment, the robot hand 300 is attached.

[0018] In this embodiment, the robot arm 200 is a vertically articulated robot arm. The base end (fixed end) of the robot arm 200 is installed on a pedestal B1. A robot hand 300 is attached to the tip (free end), which is a predetermined position of the robot arm 200.

[0019] The robot arm 200 has a base 209, multiple links 210-216, and multiple joints J1-J6. The multiple links 210-216 are connected in series via the multiple joints J1-J6 in this order. From the base end (link 210 side) of the robot arm 200 to the tip end (link 216 side) thereof, the joints are designated as a first joint J1, a second joint J2, a third joint J3, a fourth joint J4, a fifth joint J5, and a sixth joint J6. The link 210, which is the base end of the robot arm 200, is fixed to the base 209. The base 209 is fixed to the upper surface of the pedestal B1. Each of the links 211-216 is rotationally driven by each of the joints J1-J6. As a result, the robot arm 200 can adjust the robot hand 300 to any position and posture in three axial directions.

[0020] The robot hand 300 is provided to a predetermined portion, for example, a link 216 which is the tip end portion, of the robot arm 200. In other words, the link 216 is a support portion configured to support an end effector such as the robot hand 300.

[0021] The posture of the robot arm 200 can be expressed by a coordinate system. The coordinate system T0 in FIG. 1 is a coordinate system set on the base B1 to which the robot arm 200 is fixed. e is a coordinate system set in the robot hand 300. e represents the TCP (Tool Center Position). For example, the coordinate system T e is set in the robot hand 300. The coordinate system T0 and the coordinate system T e is expressed in Cartesian coordinates on three axes: X, Y, and Z.

[0022] Coordinate system T c is a coordinate system set at the center of the visual sensor 800, and the coordinate system T o and coordinate system T e Similarly, the coordinate system is expressed in three orthogonal coordinate systems consisting of the X, Y, and Z axes, and the optical axis direction of the visual sensor 800 is set to be the Z axis direction. Note that in this embodiment, the visual sensor 800 is described as being fixed to a predetermined position based on the coordinate system T0, for example, to a pedestal B1, but it may also be fixed to the robot arm 200 or the robot hand 300.

[0023] The control device 400 can control the movement, i.e., the posture, of the robot arm 200. The control device 400 is connected to a servo control unit 230, an input device 500, a display 600, and a visual sensor 800. If the input device 500 is, for example, a teaching pendant, it can be used by an operator (user) to teach the movement of the robot arm 200.

[0024] The servo control unit 230 drives and controls the motors 231 (FIG. 2) of the joints J1 to J6. The servo control unit 230 is disposed inside the base 209, for example. The location of the servo control unit 230 is not limited to inside the base 209, and the servo control unit 230 may be disposed anywhere. For example, the servo control unit 230 may be disposed inside the housing of the control device 400. That is, the servo control unit 230 may be a part of the configuration of the control device 400. In this embodiment, the input device 500, the display 600, the visual sensor 800, the control device 400, and the servo control unit 230 form a control system 440. The control system 440 is also an example of an image processing device.

[0025] The servo control unit 230 controls the driving of the motors 231 of the joints J1 to J6 so that the angle or torque of each of the joints J1 to J6 follows the command value, based on each command value corresponding to each of the joints J1 to J6 obtained from the control device 400. In other words, the servo control unit 230 is configured to be able to control the position or torque of each of the joints of the robot 100.

[0026] Fig. 2 is a block diagram showing a control system 440 of the robot device 1000 according to this embodiment. Each of the joints J1 to J6 has a motor 231, an angle sensor 250, and a torque sensor 260. Note that Fig. 2 shows the configuration of one of the joints J1 to J6 as a representative.

[0027] The control device 400 is configured by a computer and has a CPU (Central Processing Unit) 401 as a processor.

[0028] The control device 400 also includes, as examples of storage units, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, and an HDD (Hard Disk Drive) 404. The control device 400 also includes a recording disk drive 405 and a plurality of input / output interfaces (I / F) 406-410.

[0029] A ROM 402, a RAM 403, an HDD 404, a recording disk drive 405, and interfaces 406 to 410 are connected to the CPU 401 via a bus 420. Basic programs such as BIOS are stored in the ROM 402. The RAM 403 is a storage device that temporarily stores various data such as the results of calculations performed by the CPU 401.

[0030] The HDD 404 is a storage device that stores the results of arithmetic processing by the CPU 401 and various data acquired from the outside. A program 430 for causing the CPU 401 to execute arithmetic processing is recorded on this HDD 404. The CPU 401 executes an image processing method, which is a control method described below, i.e., various processes of a method for manufacturing an article, based on the program 430 recorded (stored) on the HDD 404. The recording disk drive 405 can read out various data, programs, etc. recorded on the recording disk 431.

[0031] An input device 500 is connected to the interface 406. The CPU 401 acquires input data (input information) from the input device 500 via the interface 406 and the bus 420. A display 600 is connected to the interface 407. Various images are displayed on the display 600 under the control of the CPU 401. The interface 408 is configured to be connectable to an external storage device 700, which is a storage unit such as a rewritable nonvolatile memory or an external HDD.

[0032] The servo control unit 230 is connected to the interface 409. The servo control unit 230 is connected to the motors 231, angle sensors 250, and torque sensors 260 of the joints J1 to J6 of the robot arm 200. The motors 231 are, for example, brushless DC motors or AC motors, and rotate the corresponding joints via speed reducers (not shown). The angle sensors 250 are, for example, rotary encoders, and are provided in the motors 231 and configured to be able to detect the rotation angle of the motors 231. The torque sensors 260 are provided in the corresponding joints and configured to be able to detect the torque acting on the corresponding joints.

[0033] The CPU 401 can acquire angle information from the angle sensor 250 and torque information from the torque sensor 260 via the servo control unit 230, the interface 409, and the bus 420. The servo control unit 230 may divide the angle of the motor 231 detected using the angle sensor 250 by the reduction ratio of a reducer (not shown), convert it into angle information of the corresponding joint, and transmit it to the CPU 401.

[0034] The CPU 401 outputs command value data corresponding to each of the joints J1 to J6 to the servo control unit 230 via the bus 420 and the interface 409 at predetermined time intervals (for example, 1 ms).

[0035] A visual sensor 800 is connected to the interface 410. The visual sensor 800 captures images at predetermined time intervals (e.g., 30 ms) under the control of the CPU 401. This allows the CPU 401 to obtain visual information, i.e., captured image data, from the visual sensor 800 at predetermined time intervals (e.g., 30 ms).

[0036] The HDD 404 is also a non-transitory recording medium readable by a computer. In this embodiment, the program 430 is stored in the HDD 404, but this is not a limitation. The program 430 may be recorded on any non-transitory recording medium readable by a computer. For example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, or a non-volatile memory can be used as a recording medium for providing the program 430. Examples of optical disks include disk media such as Blu-ray disks, DVDs, and CDs. Examples of non-volatile memory include storage devices such as USB memory, memory cards, ROMs, and SSDs.

[0037] Next, the working environment in this embodiment will be described. FIG. 3 is a schematic diagram showing the working environment of the robot device 1000 according to this embodiment. In this embodiment, an example will be described in which the predetermined task to be performed by the robot 100 is to assemble a workpiece W1 held by the robot hand 300 to a workpiece W2 fixed to a workpiece fixing jig M1. An area A10 indicated by a dashed line in FIG. 3 indicates an area imaged by the visual sensor 800. As shown in FIG. 3, the robot 100 can hold the workpiece W1 by the robot hand 300 holding the workpiece W1. The workpiece W1 is an example of a held object. Furthermore, the workpiece W1 is an example of a first workpiece, and the workpiece W2 is an example of a second workpiece.

[0038] 4(a) and 4(b) are schematic diagrams for explaining the assembling operation according to the embodiment. FIG. 4(a) shows the initial state of the workpiece W1. FIG. 4(b) shows the state after the workpiece W1 has been assembled to the workpiece W2. The operation of assembling the workpiece W1 to the workpiece W2 is performed by controlling the robot arm 200 so that the workpiece W1 changes from the initial state shown in FIG. 4(a) to the assembled state shown in FIG. 4(b).

[0039] Here, when teaching the state in which the assembly of the workpiece W1 is completed, the robot hand 300 is operated so that it reaches the assembly completed state shown in Fig. 4(b). The operation of the robot hand 300 may be performed by the user backdriving the robot arm 200, or the user may detach the robot hand 300 from the robot arm 200 and perform the operation directly.

[0040] FIG. 5 is an explanatory diagram of a user interface (UI) image 900 according to the embodiment. The UI image 900 is an example of a second user interface image, and is an image that accepts settings for characteristic parts, which will be described later. The CPU 401 of the control device 400 executes the program 430 to display the UI image 900 of FIG. 5 on the display screen 601 of the display 600 of FIG. 1. The UI image 900 can be operated by a user using the input device 500. The user can make various settings related to the operation of assembling the workpiece W1 to the workpiece W2 via the UI image 900.

[0041] The UI image 900 includes a camera image display section 901, a target image display checkbox 902, a current image display checkbox 903, a target image acquisition button 904, a point extraction button 905, a line extraction button 906, and a surface extraction button 907. The UI image 900 also includes a save button 908, an execute button 909, a list display section 950, a register button 951, a delete button 952, and a parameter display section 960. That is, the CPU 401 displays the camera image display section 901, the target image display checkbox 902, the current image display checkbox 903, the target image acquisition button 904, the point extraction button 905, the line extraction button 906, and the surface extraction button 907 in the UI image 900. The CPU 401 also displays the save button 908, the execute button 909, the list display section 950, the register button 951, the delete button 952, and the parameter display section 960 in the UI image 900.

[0042] Fig. 6 is a flowchart showing a process for setting setting information according to the embodiment. The CPU 401 executes the program 430 to perform the process according to the flowchart shown in Fig. 6. Fig. 7 is an explanatory diagram showing a target image according to the embodiment. Figs. 8(a) to 13(b) are explanatory diagrams of UI images according to the embodiment.

[0043] In step S101, when the user operates the target image acquisition button 904, the CPU 401 acquires a target image Ig (FIG. 7), which is an example of a predetermined image, from the visual sensor 800. The target image Ig may be acquired from the HDD 404, the external storage device 700, or a network. The target image Ig is temporarily stored in, for example, the RAM 403. The target image Ig is image data (visual information) of the completed assembly state shown in FIG. 4(b). That is, the target image Ig includes images of the robot 100 in a predetermined posture, the control target objects such as the workpiece W1, and obstacles such as the workpiece fixing jig M1. That is, the target image Ig is an image captured by the visual sensor 800 of the robot 100, the workpiece W1, and the workpiece W2 in a target posture in which the workpiece W1 is being assembled to the workpiece W2 as a predetermined posture.

[0044] 8A, the CPU 401 turns on the target image display checkbox 902 and displays the target image Ig in the camera image display section 901. The target image display checkbox 902 can also be operated by the user. The CPU 401 can switch between displaying and hiding the target image Ig in the camera image display section 901 depending on the ON / OFF state of the target image display checkbox 902.

[0045] The point extraction button 905, line extraction button 906, and surface extraction button 907 are each an example of a first button. In step S102, when the point extraction button 905, line extraction button 906, or surface extraction button 907 is operated, the CPU 401 acquires, from the target image Ig, a feature amount corresponding to the operated button. The feature amount in the image (image feature amount) is, for example, a feature portion such as a feature point, a feature line, or a feature surface.

[0046] 8(b) illustrates a case where the user operates the point extraction button 905. In step S102, when the user operates the point extraction button 905, the CPU 401 extracts feature points P(n)=P(1) to P(N) as feature amounts from the target image Ig. P) are extracted. P ) is a candidate feature part (feature candidate) to be set.

[0047] The feature points P(n) are points in the target image Ig where there is a large change in brightness or color, and can be extracted using, for example, the AKAZE algorithm. P is the number of extracted feature points. Note that the feature point extraction algorithm is not limited to AKAZE, and may be, for example, SIFT, SURF, or ORB.

[0048] Then, as shown in FIG. 8(b), the CPU 401 displays the feature points P(1) to P(N) on the target image Ig displayed on the camera image display unit 901. P ) are superimposed on the image. P ) is, for example, a rectangle.

[0049] In step S103, the CPU 401 calculates the feature points P(1) to P(N P ) is selected by the user. P ) are figures that can be selected by the user. When the user selects one of the feature point figures in the camera image display section 901, the CPU 401 highlights the selected figure as shown in FIG. 9(a), and sets the feature point corresponding to the selected figure as the feature point to be set. That is, the user can select a plurality of feature points P(1) to P(N) via the UI image 900. P ) feature points can be selected from the list. Highlighting of a figure includes, for example, displaying the figure with a thick line, displaying the figure in a color different from the figures of other feature points, etc. In FIG. 9(a), the selected feature points are feature points Pc1 and Pc2.

[0050] In step S104, the CPU 401 waits for the user to operate the registration button 951. When the user operates the registration button 951, the CPU 401 displays information about the selected feature points Pc1 and Pc2 in the list display section 950 as shown in FIG. 9(b), and also temporarily stores the information in the RAM 403. In this embodiment, registering information means temporarily storing the information in the RAM 403.

[0051] The registration information includes, for example, number information, name information, type information, controllability information, and feature descriptor information. The numbers are numbers assigned in the order of registration. The names are unique names given to registered feature points. The CPU 401 automatically sets, for example, f1 to fnr (nr is the registration number) in the order of registration. For example, "f1" is given to feature point Pc1, and "f2" is given to feature point Pc2. In the camera image display unit 901, "f1" is displayed near the figure corresponding to feature point Pc1, and "f2" is displayed near the figure corresponding to feature point Pc2. The names may be changeable by the user.

[0052] The type is, for example, point, line, or surface, and can be selected by the user from a pull-down list. If the register button 951 is operated with a feature point selected as the feature amount, the CPU 401 automatically selects "point" as the type. If the register button 951 is operated with a feature line selected as the feature amount, the CPU 401 automatically selects "line" as the type. If the register button 951 is operated with a feature surface selected as the feature amount, the CPU 401 automatically selects "surface" as the type.

[0053] The controllability is a value indicating whether the selected feature can be controlled together with the robot 100, and the user can select "possible" or "not possible" from a pull-down list. "possible" corresponds to a control object that can be moved by the control of the robot 100, such as the robot 100 or the workpiece W1 held by the robot 100. "Not possible" corresponds to an obstacle that cannot be moved by the control of the robot 100. The feature points Pc1 and Pc2 are feature values ​​associated with the workpiece W1 held by the robot hand 300. Therefore, the user selects "possible" as the controllability.

[0054] The feature descriptor is an internal parameter that the CPU 401 uses to obtain coordinates corresponding to the registered feature amount from the image acquired by the visual sensor 800 , and is not displayed in the list display section 950 .

[0055] Furthermore, the CPU 401 highlights, in the UI image 900, an image indicating a feature selected by the user from among the registered feature amounts. For example, a case will be described where the selected feature amounts are feature points Pc1 and Pc2. In the camera image display section 901, a figure to which "f1" is assigned and a figure to which "f2" is assigned are highlighted with a thick line or the like. In the list display section 950, the entire line displaying the information on feature point Pc1 and the entire line displaying the information on feature point Pc2 are each highlighted by being surrounded by a thick frame or the like.

[0056] In step S105, it is determined whether or not the registration of the features required for the assembly work has been completed, and if the registration has not been completed (S105: NO), the processes of steps S102 to S104 are repeated. If the registration has been completed (S105: YES), the process proceeds to the next step S106.

[0057] The above has described the case where an object to be controlled is registered, but since it is necessary to prevent the object to be controlled from coming into contact with an obstacle, the case where an obstacle is also registered will now be described. Here, in the example of this embodiment, the object to be controlled is the workpiece W1 or the robot 100. In the example of this embodiment, the obstacle is the workpiece fixing jig M1.

[0058] In step S102, when the user operates the line extraction button 906, the CPU 401 extracts characteristic lines L(n)=L(1) to L(N) as feature quantities from the target image Ig. L ) are extracted. L ) is a candidate feature part (feature candidate) to be set.

[0059] Then, the CPU 401 displays the characteristic lines L(1) to L(N) on the target image Ig displayed on the camera image display unit 901, as shown in FIG. 10(a). L ) is superimposed on the figure showing the characteristic lines L(1) to L(N L The figure representing the region is, for example, a rectangular frame. The frame is, for example, a dashed line. The feature line L(n) is a line indicating the boundary of the region, and is extracted, for example, by the Hough transform. L is the number of extracted feature lines.

[0060] In step S103, the CPU 401 calculates the characteristic lines L(1) to L(N L ) is selected by the user. L ) are figures that can be selected by the user. When the user selects one of the feature line figures in the camera image display section 901, the CPU 401 highlights the selected figure as shown in FIG. 10(b), and sets the feature line corresponding to the selected figure as the feature line to be set. That is, the user can select a plurality of feature lines L(1) to L(N) via the UI image 900. L ) can be selected. Highlighting of a figure includes, for example, displaying the figure in a thick line, or displaying the figure in a color different from the figures of other feature points. In FIG. 10(b), the selected feature lines are feature lines Lc1 and Lc2.

[0061] In step S104, the CPU 401 waits for the user to operate the registration button 951. When the user operates the registration button 951, the CPU 401 displays information about the selected characteristic lines Lc1 and Lc2 in the list display area 950 as shown in FIG.

[0062] Here, in the camera image display section 901, "f3" is displayed near the figure corresponding to the characteristic line Lc1, and "f4" is displayed near the figure corresponding to the characteristic line Lc2. Furthermore, since the register button 951 was operated with the characteristic line selected as the characteristic amount, the CPU 401 automatically selects "line." The characteristic lines Lc1 and Lc2 are characteristic amounts associated with the workpiece fixing jig M1. Therefore, the user selects "no" as the controllability.

[0063] When the delete button 952 is operated while a registered feature is selected, the CPU 401 disables the highlighting of the selected feature and its display in the list display section 950, and deletes the information temporarily saved in the RAM 403.

[0064] As described above, when the registration of the features necessary for the assembly work is completed in step S105, the process proceeds to step S106. Hereinafter, the registered features Pc1, Pc2, Lc1, and Lc2 will also be referred to as feature values ​​f1, f2, f3, and f4, corresponding to their names. Furthermore, when these feature values ​​f1, f2, f3, and f4 are not to be distinguished from one another, each feature value f1, f2, f3, and f4 will also be referred to as feature value f. The feature values ​​f1 and f2 in the target image Ig are first feature values ​​corresponding to the control object such as the workpiece W1, and the feature values ​​f3 and f4 in the target image Ig are second feature values ​​corresponding to the obstacle such as the workpiece fixing jig M1.

[0065] In step S106, the CPU 401 accepts, in the parameter display section 960, a user input of the dynamic characteristics (control parameters) of the virtual force acting on the feature value f registered in steps S102 to S105.

[0066] Fig. 11(b) shows examples of force control parameters input into parameter display section 960. The force control parameters include, for example, number information, information on the name of the feature that links the mechanical property, information on the type of mechanical property, and a numerical value for the type of mechanical property (first parameter or second parameter, described below). Fig. 11(b) shows, as an example, input rows for six force control parameters numbered "1" to "6."

[0067] In order to bring the workpiece W1 into the state of the target image Ig, in the input line of the force control parameter numbered "1," the user inputs "f1" as the name of the feature to which the mechanical property is linked, and inputs "IMP," which indicates impedance, i.e., attractive force, as the type of mechanical property. Also, in the input line of the force control parameter numbered "2," the user inputs "f2" as the name of the feature to which the mechanical property is linked, and inputs "IMP," which indicates impedance, i.e., attractive force, as the type of mechanical property, so that the workpiece W1 becomes the state of the target image Ig.

[0068] There are two fields, field “1” and field “2,” for inputting the names of features to be linked to mechanical properties. The above-mentioned names “f1” to “f4” can be input into fields “1” and “2.” Field “1,” for inputting the name of a feature, corresponds to a newly acquired captured image (captured image Ic, described later), and field “2,” for inputting the name of a feature, corresponds to the target image Ig. When the same name, for example, “f1,” is input into fields “1” and “2,” the feature f1 registered in the target image Ig is linked to a feature that matches the feature f1 among the features extracted from a newly acquired captured image from the visual sensor 800. Similarly, when the same name, for example, “f2,” is input into fields “1” and “2,” the feature f2 registered in the target image Ig is linked to a feature that corresponds to the feature f2 among the features extracted from a newly acquired captured image from the visual sensor 800.

[0069] Furthermore, the user inputs information for linking mechanical properties in the input lines of the force control parameters numbered "3" and "4" so that the workpiece W1 will avoid the workpiece fixing jig M1, i.e., so that the feature f1 will receive a virtual repulsive force from the feature f3 and f4. Similarly, the user inputs information for linking mechanical properties in the input lines of the force control parameters numbered "5" and "6" so that the workpiece W1 will avoid the workpiece fixing jig M1, i.e., so that the feature f2 will receive a virtual repulsive force from the feature f3 and f4.

[0070] That is, in the input row of the force control parameter numbered "3," the user inputs the name "f1" into the field "1" for inputting the name of the feature quantity, and the name "f3" into the field "2" for inputting the name of the feature quantity. Furthermore, the user inputs "repulsion," which indicates a repulsive force, as the type of mechanical property. Furthermore, in the input row of the force control parameter numbered "4," the user inputs the name "f1" into the field "1" for inputting the name of the feature quantity, and the name "f4" into the field "2" for inputting the name of the feature quantity. Furthermore, the user inputs "repulsion," which indicates a repulsive force, as the type of mechanical property.

[0071] Furthermore, in the input line of the force control parameter numbered "5," the user enters the name "f2" in the field "1" for entering the name of the feature, and the name "f3" in the field "2" for entering the name of the feature. Furthermore, the user enters "repulsion," which indicates a repulsive force, as the type of mechanical property. Furthermore, in the input line of the force control parameter numbered "6," the user enters the name "f2" in the field "1" for entering the name of the feature, and the name "f4" in the field "2" for entering the name of the feature. Furthermore, the user enters "repulsion," which indicates a repulsive force, as the type of mechanical property.

[0072] Furthermore, the user inputs a mechanical property value (first parameter or second parameter) for each of the force control parameter input lines numbered "1" to "6." When the user selects a corresponding location (input line) in the parameter display section 960, if the type of mechanical property of the selected corresponding location is "IMP," the CPU 401 displays a UI image 970 (FIG. 12(a)) corresponding to "IMP" on the display screen 601 of the display 600. Furthermore, if the type of mechanical property of the selected corresponding location is "repulsion," the CPU 401 displays a UI image 980 (FIG. 12(b)) corresponding to "repulsion" on the display screen 601 of the display 600. Each of the UI images 970 and 980 is an example of a first user interface image.

[0073] For example, the force control parameters numbered "1" and "2" have a mechanical property type of "IMP." Therefore, when the input row of the force control parameter numbered "1" or "2" is selected in the parameter display section 960, the CPU 401 displays a UI image 970 (FIG. 12(a)) on the display screen 601 of the display 600.

[0074] The UI image 970 displays a mechanical impedance model diagram 971, a spring coefficient input section 972, a damper coefficient input section 973, an OK button 974, and a cancel button 975. The user inputs a spring coefficient (K) into the spring coefficient input section 972. The user inputs a damper coefficient (D) into the damper coefficient input section 973.

[0075] When the user operates the OK button 974, the CPU 401 temporarily saves the values ​​input in the spring coefficient input section 972 and the damper coefficient input section 973 in the RAM 403 and closes the UI image 970. When the cancel button 975 is operated, the CPU 401 closes the UI image 970 without saving the values ​​in the RAM 403. The spring coefficient (K) and the damper coefficient (D) are examples of first parameters. As described above, the CPU 401 accepts the settings of the spring coefficient (K) and the damper coefficient (D) via the UI image 970.

[0076] For example, the force control parameters numbered "3" to "6" have a mechanical property type of "repulsion." Therefore, when an input row of any of the force control parameters numbered "3" to "6" is selected in parameter display section 960, CPU 401 displays UI image 980 (FIG. 12(b)) on display screen 601 of display 600.

[0077] The UI image 980 displays a graph 981 showing the relationship between distance and repulsive force, a repulsive force coefficient input section 982, an OK button 983, and a cancel button 984. In the repulsive force coefficient input section 982, the user inputs a repulsive force coefficient (R).

[0078] When the user operates the OK button 983, the CPU 401 temporarily saves the value input to the repulsion coefficient input section 982 in the RAM 403 and closes the UI image 980. When the cancel button 984 is operated, the CPU 401 closes the UI image 980 without saving the value in the RAM 403. The repulsion coefficient (R) is an example of a second parameter. As described above, the CPU 401 accepts the setting of the repulsion coefficient (R) via the UI image 980.

[0079] As described above, the dynamic characteristics of the virtual force (force control parameters) are registered in step S106. The registered numerical values ​​(first parameter and second parameter) are displayed in the "Value" field in the camera image display area 901, as shown in FIG. 13(a).

[0080] Furthermore, the CPU 401 displays a graphic representing the dynamic characteristics of the registered virtual force superimposed on the target image Ig in the camera image display unit 901. The dynamic characteristics of the virtual force for the force control parameters numbered "3" to "6" are "repulsion," which indicates a repulsive force, and therefore, as shown in Fig. 13(a), repulsion regions a1 and a2 are displayed with hatching, for example. The repulsion regions a1 and a2 indicate regions where the virtual repulsive force generated by the feature amounts f3 and f4 is, for example, 10 N.

[0081] In step S107, if the registration of the dynamic characteristics of the virtual force in the feature f is completed (S107: YES), the CPU 401 proceeds to the processing of step S108, and if not (S107: NO), the CPU 401 returns to the processing of step S106.

[0082] In step S108, when the user operates the save button 908, the CPU 401 saves information including the registered feature value f and the force control parameters as setting information PS in the HDD 404. That is, the setting information PS including the feature value f and the force control parameters is set. This completes the setting flow. Note that in this embodiment, setting information means saving information in storage such as the HDD 404.

[0083] The user can check on the UI image 900 whether the settings for the workpiece assembly operation are as intended. As shown in Fig. 13(b), when the user turns on the current image display checkbox 903, the CPU 401 displays a new captured image Ic acquired from the visual sensor 800 superimposed on the target image Ig in the camera image display section 901. For example, the target image Ig is made semi-transparent and the target image Ig and the captured image Ic are superimposed on each other and displayed in the camera image display section 901.

[0084] Here, the feature amounts f1 and f2 correspond to the workpiece W1, which is the control object, and the feature amounts f3 and f4 correspond to the workpiece fixing jig M1, which is the obstacle. The CPU 401 acquires feature amounts that match the feature amounts f1 and f2 in the newly acquired captured image Ic, for example, by pattern matching processing.

[0085] The CPU 401 displays the feature amounts f1 and f2 on the target image Ig as character images "f1g" and "f2g" along with a square shape on the camera image display unit 901. Hereinafter, the feature amounts f1 and f2 on the target image Ig will be referred to by the same symbols as the character images "f1g" and "f2g", i.e., feature amounts f1g and f2g. As described above, feature amounts f1g and f2g are an example of a first feature portion. The target image Ig and feature amounts f1g, f2g, f3, f4, etc. are pre-set information.

[0086] Similarly, the CPU 401 displays feature amounts that match the feature amounts f1g and f2g on the captured image Ic as character images "f1c" and "f2c" along with a square shape on the camera image display unit 901. Hereinafter, the feature amounts that match the feature amounts f1g and f2g on the captured image Ic will be referred to as feature amounts f1c and f2c, which are the same symbols as the character images "f1c" and "f2c." The feature amounts f1c and f2c are an example of a third feature portion.

[0087] The feature quantities f1g and f1c are associated with each other by a virtual impedance characteristic (virtual attractive force) according to the setting information PS, and the feature quantities f2g and f2c are associated with each other by a virtual impedance characteristic (virtual attractive force) according to the setting information PS. That is, the CPU 401 sets (defines) the virtual attractive force acting between the feature quantities f1g and f1c based on the setting information PS, and sets (defines) the virtual attractive force acting between the feature quantities f2g and f2c based on the setting information PS. The CPU 401 displays the virtual attractive force between the feature quantities f1g and f1c as a figure s1 on the camera image display unit 901. The CPU 401 also displays the virtual attractive force between the feature quantities f2g and f2c as a figure s2 on the camera image display unit 901. The figures s1 and s2 are figures that allow the user to intuitively understand that they are attractive forces, such as triangular wave figures, i.e., spring figures. The figure s1 is a figure connecting the figure representing the feature quantity f1g and the figure representing the feature quantity f1c. The figure s2 is a figure connecting the figure representing the feature quantity f2g and the figure representing the feature quantity f2c. Hereinafter, the virtual attractive force between the feature quantity f1g and the feature quantity f1c will be referred to as the attractive force s1, which has the same sign as the figure s1. The virtual attractive force between the feature quantity f2g and the feature quantity f2c will be referred to as the attractive force s2, which has the same sign as the figure s1. The attractive force s1 is obtained using the parameters set in the force control parameter number "1", and the attractive force s2 is obtained using the parameters set in the force control parameter number "2".

[0088] Furthermore, the feature quantity f3 and the feature quantity f1c are associated by a virtual repulsive force based on the setting information PS, and the feature quantity f4 and the feature quantity f1c are associated by a virtual repulsive force based on the setting information PS. The feature quantity f3 and the feature quantity f2c are associated by a virtual repulsive force based on the setting information PS, and the feature quantity f4 and the feature quantity f2c are associated by a virtual repulsive force based on the setting information PS. That is, the CPU 401 sets (defines) the virtual repulsive force acting between the feature quantity f3 and the feature quantity f1c based on the setting information PS, and sets (defines) the virtual repulsive force acting between the feature quantity f4 and the feature quantity f1c based on the setting information PS. Similarly, the CPU 401 sets (defines) the virtual repulsive force acting between the feature quantity f3 and the feature quantity f2c based on the setting information PS, and sets (defines) the virtual repulsive force acting between the feature quantity f4 and the feature quantity f2c based on the setting information PS. The calculation method for these virtual forces (virtual attractive force and virtual repulsive force) will be described later.

[0089] The user can instruct the control device 400 to start the assembly work from the UI image 900 or the input device 500. When using the UI image 900, the user can instruct the start of the work by operating the execute button 909. When using the input device 500, the user can instruct the start of the work by specifying the setting information PS with the input device 500.

[0090] FIG. 14 is a flowchart showing force control for performing assembly work according to the embodiment. When an instruction to start work is given, the CPU 401 and the servo control unit 230 execute processing in accordance with the flowchart shown in FIG. 14. FIGS. 15(a) and 15(b) are control block diagrams according to the embodiment. Here, the CPU 401 functions as the feedback control unit 450 shown in FIG. 15(a) by executing the program 430. In this embodiment, the feedback control unit 450 and the servo control unit 230 are examples of a control unit (processing unit), and the robot 100 can be force-controlled by their cooperation.

[0091] In this embodiment, a minor loop is configured in which the torque detection value τ of the torque sensor 260 is fed back to the servo control unit 230. Also, a major loop is configured in which the captured image Ic of the visual sensor 800 and the angle detection value q of the angle sensor 250 are fed back to the feedback control unit 450. This major loop performs full-closed loop control to control the robot 100 so that the error of the captured image Ic relative to the target image Ig is reduced.

[0092] The initial state of the workpiece W1 at which force control is initiated is assumed to be the state shown in FIG. 4(a). That is, until the state shown in FIG. 4(a) is reached, the control device 400 and the servo control unit 230 control the robot 100 by position control. The position control is based on an angle command value and an angle detection value q. The position control brings the angle detection value q closer to the angle command value. When the state shown in FIG. 4(a) is reached, the control device 400 and the servo control unit 230 start force control of the robot 100.

[0093] In step S20, the feedback control unit 450 reads the target image Ig, setting information PS, and model information MO stored in the HDD 404. The model information MO includes information on link parameters used in the kinematics calculation and dynamics calculation of the robot arm 200, as well as information on the dynamics model.

[0094] In step S21, the feedback control unit 450 acquires from the visual sensor 800 the captured image Ic that the visual sensor 800 has generated by capturing an image.

[0095] Furthermore, in step S22, the feedback control unit 450 acquires the detected angle value (joint angle) q of each of the joints J1 to J6 of the robot arm 200 via the servo control unit 230. Note that the joint angle q is an angle value obtained by converting the angle acquired by the angle sensor 250 based on the reduction ratio of a reducer (not shown) or the like, but in Fig. 15(a) the conversion unit that performs this conversion process is not shown. The function of this conversion unit may be included in the control device 400 or the servo control unit 230.

[0096] In step S23, the feedback control unit 450 calculates the torque command value τ d A block diagram of the processing in the feedback control unit 450 is shown in FIG.

[0097] As shown in FIG. 15( b ), the feedback control unit 450 includes a feature extraction unit 451 , a virtual force calculation unit 452 , a filter processing unit 453 , a Jacobian calculation unit 454 , a gravity compensation torque calculation unit 455 , and a torque command value calculation unit 456 .

[0098] The feature extraction unit 451 extracts the feature f set in the setting information PS from each of the target image Ig and the captured image Ic based on the feature descriptor. For example, as shown in FIG. 13(b), the feature extraction unit 451 extracts feature amounts f1g, f2g, f3, and f4 from the target image Ig and extracts feature amounts f1c and f2c from the captured image Ic. Note that since the feature amounts f1g, f2g, f3, and f4 have already been set, loaded information (data) may be used. As described above, the feature amounts f1g and f2g are an example of a first feature portion, the feature amounts f3 and f4 are an example of a second feature portion, and the feature amounts f1c and f2c are an example of a third feature portion.

[0099] The feature extraction unit 451 extracts the difference f of the feature in which the dynamic relationship of the virtual force is set. e (f is in bold) vector. For example, the difference f of the feature corresponding to the force control parameter number "1" is calculated. e1 The vector (f is in bold) is calculated according to the following formula (1).

number

[0100] where bold is a vector or matrix. 1c The vector (f is in bold) is the position on the image of the feature extracted from the captured image Ic ([u 1c v 1c ] T ) and f1g The vector (f is in bold) represents the position on the image of the feature extracted from the target image Ig ([u 1g v 1g ] T ) The superscript "T" denotes the transpose of a matrix or vector.

[0101] The feature extraction unit 451 performs the same calculation for the differences in the feature values ​​corresponding to the force control parameters with other numbers included in the setting information PS. In the example of FIG. 13(b), since there are six force control parameters, the difference f e The vector (f is in bold) is a 12-dimensional vector, f e =[f e1 T … f e6 T ] T is.

[0102] The feature extraction unit 451 extracts the difference f e The vector is passed to the virtual force calculation unit 452. When the associated feature amounts are a combination of a point and a line, such as the force control parameters numbered "3" to "6", the shortest distance between them is calculated as the difference between the feature amounts.

[0103] The virtual force calculation unit 452 calculates the feature amount difference f e , and the virtual force F acting between each feature based on the force control parameters included in the setting information PS. v (F is in bold)

[0104] For example, the virtual force F corresponding to the force control parameter number "1" in which the virtual impedance characteristic is set v1 The vector (F is in bold) is calculated according to the following formula (2).

number

[0105] In formula (2), K d1 ,D d1are scalar values ​​representing the spring coefficient and damper coefficient in the virtual impedance characteristic corresponding to the force control parameter numbered "1."

[0106] In addition, the virtual force F corresponding to the force control parameter number "3" in which the virtual repulsive force characteristic is set is v3 The vector (F is in bold) is calculated according to the following formula (3).

number

[0107] In equation (3), R3 is a coefficient that relates the distance to the repulsive force in the virtual repulsive force corresponding to the force control parameter numbered "3."

[0108] The feature extraction unit 451 performs the same calculation for the virtual forces corresponding to the force control parameters with other numbers included in the setting information PS. In the example of FIG. 13(b), since there are six force control parameters, the virtual forces F v The vector (F in bold) is a 12-dimensional vector, F v =[F v1 T … F v6 T ] T The feature extraction unit 451 extracts the virtual force F v The vector is passed to the filter processing unit 453.

[0109] In this way, the feature extraction unit 451 compares the target image Ig with the captured image Ic to obtain the virtual force F as information about the force. v That is, the feature extraction unit 451 obtains a vector of the virtual attractive force F based on the relationship between the feature f1c and the feature f1g. v1 , the virtual attractive force F based on the relationship between the feature f2c and the feature f2g v2 The feature extraction unit 451 also calculates a virtual repulsive force F based on the relationship between the feature f1c and the feature f3. v3 , the virtual repulsive force F based on the relationship between the feature f1c and the feature f4 v4 , the virtual repulsive force F based on the relationship between the feature f2c and the feature f3v5 , the virtual repulsive force F based on the relationship between the feature f2c and the feature f4 v6 Ask for.

[0110] The filter processing unit 453 calculates the virtual force F v A predetermined filter process is performed on each element of the vector. The filter used for the filter process is a digital filter obtained by discretizing the transfer function shown in the following equation (4) using a bilinear transform or the like.

number

[0111] This transfer function is a second-order notch filter, which has the effect of reducing the gain in a specific frequency range and is used as a method for stabilizing control. Here, s is the differential operator, ω n is the center frequency of the notch, ζ is the width of the notch, and d is the variable that determines the depth of the notch. The filtered virtual force F v is passed to the torque command value calculation unit 456.

[0112] The Jacobian calculation unit 454 calculates the image Jacobian J for each feature value f. img (J is in bold) matrix and the robot Jacobian J r (J is in bold) Calculate the image Jacobian J img is the coordinate system T set in the robot hand 300. e It is an 8-by-6 matrix that associates the small displacement of the feature f with the small displacement of the robot Jacobian J r is the small displacement of each joint J1 to J6 of the robot arm 200 and the coordinate system T e is a 6-by-6 matrix that associates the small displacement of the image with the img and the robot Jacobian J r Define

number

[0113] where xe (x in bold) is the coordinate system T in the coordinate system To e The six-degree-of-freedom position vector x e =[X e Y e Z e α e β e gamma e ] T q (q is in bold) is the joint angle vector q=[q1...q6] of each joint J1 to J6 of the robot arm 200. T is.

[0114] The Jacobian calculation unit 454 calculates the image Jacobian J img and the robot Jacobian J r The integrated Jacobian J (J is in bold) with 8 rows and 6 columns is calculated by taking the inner product of the above. In other words, the integrated Jacobian J is a matrix that associates the motion of the robot arm 200 with the motion of the feature quantities.

[0115] The gravity compensation torque calculation unit 455 calculates a gravity compensation torque τ that balances with the estimated value of the gravity torque generated by gravity at each of the joints J1 to J6 of the robot arm 200 based on the model information MO and the joint angle q of each of the joints J1 to J6 of the robot arm 200. g (τ is in bold) Calculate the gravity compensation torque τ g is calculated by deriving the equations of motion using, for example, the Newton-Euler method.

[0116] The torque command value calculation unit 456 calculates the filtered virtual force F v , the integrated Jacobian J, and the gravity compensation torque τ g Based on this, the torque command value τ d (τ is in bold)

number

[0117] In this way, the torque command value calculation unit 456 calculates the virtual force F v Based on this, the torque command value τ for force control of the robot 100 is calculated. d Get.

[0118] In step S24, the feedback control unit 450 calculates the difference f e It is determined whether each element included in is equal to or less than a predetermined threshold. If all elements are equal to or less than the predetermined threshold (S24: YES), the feedback control unit 450 and the servo control unit 230 end the process. If not (S24: NO), the feedback control unit 450 proceeds to the process of step S25. For example, the predetermined threshold is set to 3 pixels for the force control parameters numbered "1" and "2". Note that pixel represents the unit per pixel of the image.

[0119] In step S25, the feedback control unit 450 acquires the detected torque value τ from the torque sensor 260 of each of the joints J1 to J6 of the robot arm 200.

[0120] In step S26, the servo control unit 230 determines whether the torque detection value τ is equal to the torque command value τ d In accordance with the above, the servo control unit 230 calculates a current command value for the current to be supplied to the motor 231 of each of the joints J1 to J6 of the robot arm 200. Then, the servo control unit 230 supplies the current i d That is, the servo control unit 230 drives the motor 231 by supplying the torque value τ and the torque command value τ d Torque control is performed based on the difference between the

[0121] After completing the process of step S26, the feedback control unit 450 returns to the process of step S21. In this way, the processes of steps S21 to S26 are repeatedly executed every time a new captured image Ic is acquired by the feedback control unit 450. Through the above control process, the control device 400 and the servo control unit 230 can control the robot arm 200 based on the virtual mechanical characteristics set by the user in the feature space.

[0122] Next, the process of assembling the workpiece W1 to the workpiece W2 using the setting information PS will be described in detail. Figures 16(a) to 17(c) are explanatory diagrams of the camera image display section 901 that displays the state of the assembling work according to the embodiment.

[0123] First, when the workpiece W1 is in the initial state, the positional relationship between the workpiece W1, the workpiece W2, the workpiece fixing jig M1, and the feature value f associated with them is as shown in FIG. 16(a).

[0124] When the control device 400 starts controlling the robot arm 200, the feature quantities f1c and f1g attract each other, and the feature quantities f2c and f2g attract each other due to the action of the virtual impedance characteristics corresponding to the force control parameters numbered "1" and "2." Therefore, as shown in FIG. 16(b), the feature quantity f1c approaches the feature quantity f1g, and the feature quantity f2c approaches the feature quantity f2g.

[0125] As shown in FIG. 16(c), when feature f1c approaches repulsion region a1, the virtual repulsive force characteristics corresponding to the force control parameter number "3" cause workpiece W1 to behave as if it were in contact with an invisible wall before contacting workpiece fixing jig M1. At this time, the virtual impedance characteristics corresponding to the force control parameters numbered "1" and "2" cause workpiece W1 and workpiece W2 to continue to attract each other. Therefore, as shown in FIG. 17(a), workpiece W1 slides along the outer edge of repulsion region a1 so that feature f1c approaches feature f1g and feature f2c approaches feature f2g. This prevents workpiece W1 from colliding with workpiece fixing jig M1.

[0126] As shown in FIG. 17(b), when the feature value f1c passes through the repulsion area a1, the workpiece W1 moves in the coordinate system T o Then, as shown in FIG. 17(c), the tip of the workpiece W1 comes into contact with the opening of the workpiece W2.

[0127] Then, due to the action of the virtual impedance characteristics corresponding to the force control parameters numbered "1" and "2", the workpiece W1 follows the shape of the workpiece W2, and the difference f e1 and f e2 When all of the components of the above become 3 pixels or less, the control device 400 ends the assembling work. In this way, an article is manufactured in which the workpiece W1 is assembled to the workpiece W2.

[0128] As described above, the feedback control unit 450 and the servo control unit 230 execute force control based on the difference between the target image Ig and the captured image Ic, causing the posture of the robot 100 to approach the target posture when the target image Ig was captured. That is, the position and posture of the workpiece W1 approach the position and posture in the target image Ig. Then, when the posture of the robot 100 roughly matches the target posture, the assembly of the workpiece W1 is completed.

[0129] As described above, according to this embodiment, the robot 100 is controlled so that virtual forces act between the feature amounts f1c and f2c extracted from the captured image Ic and the set feature amounts f1g, f2g, f3, and f4. Therefore, a force-based task can be set in the feature amount space, and the workability of the robot 100 can be improved.

[0130] For example, virtual attractive forces can be set for the feature quantities f1g and f2g associated with the workpiece W1, which is the control target, and virtual repulsive forces can be set for the feature quantities f3 and f4 associated with the workpiece fixing jig M1, which is the obstacle. This makes it possible to perform both contact-related tasks such as assembly work and other operations such as avoiding obstacles. This reduces the chance of the workpiece W1 getting caught on an obstacle, such as the workpiece fixing jig M1, and reduces the frequency of assembly failures, thereby improving the workability of the robot 100.

[0131] The present invention is not limited to the above-described embodiments, and many modifications are possible within the technical concept of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0132] In the above embodiment, the feedback control unit 450 is part of the functions of the CPU 401, and the servo control unit 230 is configured as a device different from the CPU 401. However, this is not limiting. The CPU 401 may be configured to realize some or all of the functions of the servo control unit 230 based on the program 430.

[0133] In the above-described embodiment, the control device 400 and the display 600 are used to display the UI image 900 and accept user input, but the present invention is not limited to this. For example, a separate electronic device including a CPU and a display device such as a display may be used. The electronic device may be an information processing device such as a desktop personal computer (PC), a laptop PC, a tablet PC, or a smartphone. Furthermore, if the input device 500 is a teaching pendant having a display device, the UI image may be displayed on the display device.

[0134] In the above embodiment, the robot arm 200 is a vertically articulated robot arm, but the present invention is not limited to this. The robot arm 200 may be various robot arms, such as a horizontally articulated robot arm, a parallel link robot arm, or a Cartesian robot.

[0135] In the above embodiment, the case where the robot hand 300 is attached to the robot arm 200 has been described, but the present invention is not limited to this. A holding mechanism capable of holding an object such as a workpiece may be attached to the robot arm 200 as an end effector. Examples of the holding mechanism include a mechanism that holds a workpiece by suction. In addition, a tool for processing the workpiece may be attached to the robot arm 200 as an end effector.

[0136] Furthermore, although a robot is used in the above-described embodiment, the present invention is not limited to this. For example, the present invention can also be applied to a machine that can automatically perform movements such as extension and contraction, bending and stretching, vertical movement, horizontal movement, or rotation, or a combination of these movements, based on information stored in a storage device provided in the control device.

[0137] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0138] The disclosure of the above embodiments includes the following configurations and methods.

[0139] (Configuration 1) Robots and An imaging unit; a control unit that controls the robot, the control unit compares a predetermined image with an image of the robot captured by the imaging unit to obtain information regarding force, and performs force control on the robot based on the information regarding force. A robot device characterized by:

[0140] (Configuration 2) the predetermined image is an image of the robot in a predetermined posture, the control unit causes the posture of the robot to approach the predetermined posture through the force control. 2. The robot device according to configuration 1.

[0141] (Configuration 3) the predetermined image includes a first feature portion corresponding to a control object and a second feature portion corresponding to an obstacle, The control unit acquiring a third feature portion corresponding to the control object from the captured image; acquiring information about the force based on a relationship between the third feature portion and the first feature portion and a relationship between the third feature portion and the second feature portion; 3. The robot device according to configuration 1 or 2.

[0142] (Configuration 4) The control object is the robot or an object held by the robot. 4. The robot device according to configuration 3.

[0143] (Configuration 5) the control unit sets, as the information regarding the force, a virtual attractive force between the third feature portion and the first feature portion, and a virtual repulsive force between the third feature portion and the second feature portion. 5. The robot device according to configuration 3 or 4.

[0144] (Configuration 6) the control unit sets the virtual attractive force using a first parameter and sets the virtual repulsive force using a second parameter; 6. The robot device according to configuration 5.

[0145] (Configuration 7) the control unit displays, on a display unit, a first user interface image that accepts settings of the first parameter and the second parameter. 7. The robot device according to configuration 6.

[0146] (Configuration 8) the control unit displays, on a display unit, a second user interface image that accepts settings of the first feature part and the second feature part. 8. The robot device according to any one of configurations 3 to 7.

[0147] (Configuration 9) the control unit displays the predetermined image in the second user interface image. 9. The robot device according to configuration 8.

[0148] (Configuration 10) the control unit displays, in the second user interface image, buttons for acquiring a plurality of feature candidates that are candidates for the first feature portion and the second feature portion from the predetermined image. 10. The robot device according to configuration 9.

[0149] (Configuration 11) The control unit when the button is operated, graphics corresponding to the plurality of feature candidates are displayed on the predetermined image displayed on the second user interface image so that the user can select the first feature portion and the second feature portion to be set from the plurality of feature candidates. 11. The robot device according to configuration 10.

[0150] (Configuration 12) The control unit displaying figures corresponding to the set first characteristic portion and the set second characteristic portion on the predetermined image displayed on the second user interface image; 12. The robot device according to any one of configurations 9 to 11.

[0151] (Configuration 13) the control unit displays the predetermined image and the captured image in a superimposed manner in the second user interface image. 13. The robot device according to any one of configurations 8 to 12.

[0152] (Method 14) A method for controlling a robotic device, comprising: acquiring information about the force by comparing the predetermined image with the image of the robot captured by the imaging unit; force-controlling the robot based on information about the force; A method for controlling a robot device.

[0153] (Configuration 15) An image processing device that acquires information about a force for force control of a robot, a control unit that acquires information about the force by comparing a predetermined image with an image of the robot captured by an imaging unit; 1. An image processing device comprising:

[0154] (Method 16) An image processing method for acquiring information about forces for force control of a robot, comprising: comparing the predetermined image with the image of the robot captured by the imaging unit; obtaining information about the force; An image processing method comprising:

[0155] (Method 17) 14. A method for manufacturing an article, characterized in that the robot device according to any one of configurations 1 to 13 is used to manufacture an article.

[0156] (Configuration 18) A program for causing a computer to execute the method for controlling a robot device according to Method 14.

[0157] (Configuration 19) A computer-readable recording medium having the program according to configuration 18 recorded thereon. [Explanation of symbols]

[0158] Ic...captured image, Ig...target image (predetermined image), 100...robot, 230...servo control unit (control unit), 450...feedback control unit (control unit), 800...visual sensor (imaging unit), 1000...robot device

Claims

1. Robots and An imaging unit; a control unit that controls the robot, The control unit acquiring information about virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on the target image and the current image captured by the imaging unit; controlling the robot based on the information about the mechanical characteristics; A robot system characterized by:

2. A virtual attractive force and a virtual repulsive force are set in a specific part of the target image, acquiring the current image in which a feature portion corresponding to a feature portion of the target image is captured using the imaging unit; the control unit controls the robot to avoid the characteristic part in the target image to which the virtual repulsive force is set when the characteristic part in the current image approaches the characteristic part in the target image to which the virtual attractive force is set.

2. The robot system according to claim 1.

3. The control unit displays at least one piece of information on the type of the acquired mechanical property and the parameter of the mechanical property on a display unit.

2. The robot system according to claim 1.

4. The control unit allows a user to set whether or not to superimpose the target image and the current image.

2. The robot system according to claim 1.

5. the target image is an image of the robot in a predetermined posture, the control unit controls the robot so that the posture of the robot approaches the predetermined posture.

2. The robot system according to claim 1.

6. the target image includes a first feature portion corresponding to a control object and a second feature portion corresponding to an obstacle; The control unit acquiring a third feature portion corresponding to the control object from the current image; acquiring the mechanical characteristics acting between the third characteristic portion and the first characteristic portion, and the mechanical characteristics acting between the third characteristic portion and the second characteristic portion; 2. The robot system according to claim 1.

7. The control object is the robot or an object held by the robot.

7. The robot system according to claim 6.

8. the control unit acquires a virtual attractive force as the mechanical property acting between the third feature portion and the first feature portion, and acquires a virtual repulsive force as the mechanical property acting between the third feature portion and the second feature portion.

7. The robot system according to claim 6.

9. The control unit displays a virtual attractive force acting between the third feature portion and the first feature portion in the target image or the current image using a first figure, and displays a virtual repulsive force acting between the third feature portion and the second feature portion in a second figure different from the first figure.

7. The robot system according to claim 6.

10. the control unit acquires the virtual attractive force using a first parameter and acquires the virtual repulsive force using a second parameter; 3. The robot system according to claim 2.

11. the control unit displays, on a display unit, a first user interface image that accepts settings of the first parameter and the second parameter. The robot system according to claim 10 .

12. the control unit displays, on the display unit, a second user interface image that accepts setting of a characteristic part of the target image.

2. The robot system according to claim 1.

13. the control unit displays the target image in the second user interface image. The robot system according to claim 12 .

14. the control unit displays, in the second user interface image, a user interface for acquiring a plurality of feature candidates that are candidates for feature portions from the target image. The robot system according to claim 13 .

15. The control unit when the user interface is operated, a graphic corresponding to each of the plurality of feature candidates is displayed on the target image displayed on the second user interface image so that the user can select a feature portion to be set from the plurality of feature candidates; The robot system according to claim 14 .

16. The control unit displaying graphics corresponding to the set characteristic portions on the target image displayed on the second user interface image; The robot system according to claim 12 .

17. the control unit displays the target image and the current image in an overlapping manner in the second user interface image. The robot system according to claim 12 .

18. A method for controlling a robot system, comprising: acquiring information about virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on the target image and the current image captured by an imaging unit; controlling the robot based on the information about the mechanical characteristics; A control method comprising:

19. An image processing device that acquires information for controlling a robot, a processing unit that acquires information about virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on the target image and the current image captured by the imaging unit; 1. An image processing device comprising:

20. An image processing method for acquiring information for controlling a robot, comprising: acquiring information about virtual mechanical properties acting between a feature portion in the target image and a feature portion in the current image based on the target image and the current image captured by an imaging unit; An image processing method comprising:

21. A method for manufacturing an article, comprising manufacturing an article using the robot system according to any one of claims 1 to 17.

22. A program for causing a computer to execute the control method according to claim 18 or the image processing method according to claim 20.

23. A computer-readable recording medium on which the program according to claim 22 is recorded.

Images