Robot control device and robot system

Abstract

Provided is a robot control device capable of facilitating the work of setting a control center for controlling the operation of a robot.　The robot control device controls a robot manipulator (10) which is equipped with an end effector, the robot control device comprising: an image processing unit that, by using a feature extraction model for detecting images of the robot manipulator, and position / posture information of the robot manipulator (10), detects, from images (M1, M2) in which at least part of the robot manipulator (10) is captured, a position in a three-dimensional space which corresponds to designated positions (P1, P2) designated on the images, as a position relative to the robot manipulator (10); and a coordinate system determination unit that sets a control center (101), for controlling the operation of the robot manipulator, to the position detected in the three-dimensional space.

Description

Technical Field

[0001] The present invention relates to a robot control device and a robot system.

Background Art

[0002] When constructing a robot system for causing a robot manipulator having a tool as an end effector mounted on an arm tip to execute a predetermined operation, it is necessary to set a coordinate system for controlling the operation of the robot manipulator such as a tool coordinate system (see, for example, Patent Document 1).

[0003] When setting the tool coordinate system, the user needs to perform an operation of causing the robot manipulator to take a posture from several directions so that the control center of the tool (generally the tool tip point) coincides with the reference position. Such an operation may be referred to as touch-up. For example, Patent Document 2 describes the setting of the coordinate system by touch-up.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The above-described touch-up is a highly skilled and time-consuming operation, and is particularly likely to cause mistakes for beginners who are not used to it. A robot control device and a robot system that can facilitate the setting operation of the control center for controlling the operation of the robot are desired.

Means for Solving the Problems

[0006] One aspect of the present disclosure is a robot control device for controlling a robot manipulator equipped with an end effector, comprising: a feature extraction model for image detection of the robot manipulator; an image processing unit that uses position and orientation information of the robot manipulator to detect a position in three-dimensional space corresponding to a specified position on an image of at least a part of the robot manipulator as a relative position to the robot manipulator; and a coordinate system determination unit that moves a flange coordinate system position set at the flange center of the robot manipulator (which has been set in advance) to the position detected in the three-dimensional space, thereby setting a control center for controlling the operation of the robot manipulator to the position detected in the three-dimensional space. The image processing unit uses two or more images, each of which is an image showing at least a part of the robot manipulator and which is an image of at least a part of the robot manipulator taken from different directions, to detect the position in the three-dimensional space corresponding to a specified position specified in each of the two or more images. This is a robot control device.

[0007] Another aspect of the present disclosure includes a robot manipulator equipped with an end effector, a robot control device for controlling the robot manipulator, a teaching operation device connected to the robot control device, the teaching operation device being equipped with an imaging device, a storage unit for storing a feature extraction model for image detection of the robot manipulator, an image processing unit that uses the feature extraction model and position and orientation information of the robot manipulator to detect a position in three-dimensional space corresponding to a specified position on the image as a relative position to the robot manipulator from an image captured by the imaging device in which at least a part of the robot manipulator is captured, and a coordinate system determination unit that moves a flange coordinate system position set at the flange center of the robot manipulator in advance to the position detected in the three-dimensional space, thereby setting a control center for controlling the operation of the robot manipulator to the position detected in the three-dimensional space. The image processing unit uses two or more images, each of which is an image showing at least a part of the robot manipulator and which is an image of at least a part of the robot manipulator taken from different directions, to detect the position in the three-dimensional space corresponding to a specified position specified in each of the two or more images. It is a robotic system. [Effects of the Invention]

[0008] The above configuration makes it easier to set up the control center for controlling the robot's movements.

[0009] These and other objects, features, and advantages of the present invention will become even clearer from the detailed description of typical embodiments of the present invention shown in the accompanying drawings. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram shows the equipment configuration of a robot system according to one embodiment. [Figure 2] This is a block diagram showing the functional configuration of the robot control device and the teaching / operation device. [Figure 3] This is a flowchart illustrating the process of setting the control center and coordinate system. [Figure 4] This figure shows an example of an image captured with a detection marker. [Figure 5] This diagram illustrates an example of the shooting direction for two images taken from the tip of the arm. [Figure 6] This diagram illustrates the process of translating the flange coordinate system to set the tool coordinate system. [Figure 7] This figure shows an example of a captured image used for pattern matching, etc. [Figure 8] This diagram illustrates the process of a robotic manipulator performing cylindrical fitting. [Figure 9] This diagram shows the state in which control centers are set for mating workpieces of different lengths. [Figure 10] This diagram shows a force sensor positioned between the flange and the hand. [Figure 11] This diagram illustrates a first example of setting the pressing direction for force control. [Figure 12] This diagram illustrates a second example of setting the pressing direction for force control. [Figure 13] This diagram illustrates the operation of specifying the region in which force-controlled operation is possible. [Figure 14] This diagram illustrates how to set the tool coordinate system for a hand that consists of multiple chuck hands integrated into one unit.

Best Mode for Carrying Out the Invention

[0011] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings to be referred to, the same constituent parts or functional parts are denoted by the same reference numerals. For ease of understanding, the scales of these drawings are appropriately changed. Also, the forms shown in the drawings are one example for implementing the present invention, and the present invention is not limited to the illustrated forms.

[0012] FIG. 1 is a diagram showing the equipment configuration of a robot system 100 according to an embodiment. As shown in FIG. 1, the robot system 100 includes a robot manipulator 10 (hereinafter referred to as robot 10) equipped with a hand 5 as an end effector, a robot control device 20 that controls the robot 10, and a teaching operation device 30 connected to the robot control device 20. The connection between the teaching operation device 30 and the robot control device 20 may be made by wired communication or wireless communication.

[0013] In this embodiment, the robot 10 is assumed to be a 6-axis vertical articulated robot, but other types of robots may be used. The hand 5 is attached to the flange surface at the tip of the arm of the robot 10.

[0014] The robot control device 20 controls the operation of the robot 10 according to an operation input from the teaching operation device 30 or according to an operation program stored in the robot control device 20. Note that the robot control device 20 may have a configuration as a general computer having a CPU, ROM, RAM, storage device, operation unit, display unit, input / output interface, network interface, and the like.

[0015] The teaching operation device 30 has functions for teaching operations to the robot 10 and for making various settings related to teaching. In Figure 1, a tablet terminal is used as the teaching operation device 30, but a teaching control panel, a smartphone, or other various portable terminals can be used as the teaching operation device 30. The teaching operation device 30 may also have a configuration as a general computer, including a CPU, ROM, RAM, storage device, operation unit, display unit, input / output interface, network interface, etc.

[0016] The teaching operation device 30 according to this embodiment further includes a camera 34 as an imaging device (see Figure 2). As will be described in detail below, the robot control device 20 according to this embodiment provides a function to set the control center and coordinate system (tool coordinate system, force control coordinate system, etc.) that are the targets of controlling the movement of the robot 10, using an image captured by the camera 34 of the teaching operation device 30 so as to include a part of the robot 10 (for example, the tip of the tool).

[0017] Figure 2 is a block diagram showing the functional configuration of the robot control device 20 and the teaching operation device 30. As shown in Figure 2, the robot control device 20 includes a motion control unit 21, an image processing unit 22, and a coordinate system determination unit 23. The robot control device 20 also stores a feature extraction model 24 in its memory unit. The motion control unit 21 performs motion control of the robot 10. More specifically, the motion control unit 21 interprets motion commands to calculate position commands for each axis motor of the robot 10, and performs position control of the control center (e.g., the tool tip) of the robot 10 by executing servo control of each axis motor according to the position commands. The motion control unit 21 may also have a function to control the opening and closing operation of the hand 5. The image processing unit 22 uses the feature extraction model 24 for image detection of the robot 10 and the position and orientation information of the robot 10 to detect the position in three-dimensional space corresponding to a specified position on the image as a relative position to the robot 10, from an image showing at least a part of the robot 10. The coordinate system determination unit 23 sets the control center for controlling the movement of the robot 10 to a position detected in three-dimensional space.

[0018] The robot 10 may also be equipped with a force sensor 3 to perform force control. Figure 2 shows an example configuration in which the robot 10 is equipped with a force sensor 3 to detect external forces applied to the tip of the arm. The force sensor 3 is positioned between the hand 5 and the flange 11, for example, as shown in Figure 10. When the robot 10 is configured to be equipped with a force sensor 3, the motion control unit 21 further includes a force control unit 25 that performs force control using the force sensor 3.

[0019] The teaching operation device 30 includes a control unit 31 that controls teaching input operations and various setting functions, a display unit 32 that displays operation screens related to teaching input operations and various setting functions, and an operation unit 33 for performing various input operations. The display unit 32 has, for example, a liquid crystal display as a display device. The operation unit 33 may consist of software keys using a touch panel, for example. The teaching operation device 30 further includes a camera 34 as an imaging device. The camera may be, for example, a two-dimensional camera that captures two-dimensional images, or a three-dimensional camera that acquires the three-dimensional position of an object, for example, by stereo. Images captured by the camera 34 are displayed on the display unit 32. As will be described later, the user can set the control center and coordinate system for controlling the movement of the robot 10 by specifying the position of the control center on the image obtained by photographing the robot 10 using the camera 34 of the teaching operation device 30. The teaching operation device 30 operates under the control of the image processing unit 22 of the robot control device 20, displaying the captured image on the display unit 32, and then accepting the above-mentioned specified operations via the operation unit 33.

[0020] Next, we will explain the function of setting the control center and coordinate system by having the user specify the control center on the image captured by the camera 34 of the teaching operation device 30 (hereinafter referred to as the control center and coordinate system setting process). Figure 3 is a flowchart showing the flow of the control center and coordinate system setting process. The control center and coordinate system setting process is mainly executed under the control of the CPU of the robot control device 20, and the teaching operation device 30 is used as an operation terminal for the user to input various operations.

[0021] First, the user takes two images of a part of the robot 10 (or the detection markers) from different angles (step S1). The two images taken are referred to as camera image M1 and camera image M2. Camera image M1 is, for example, the image shown in Figure 4. The example of camera image M1 in Figure 4 shows the tip of the robot arm, the flange 11, and the hand 5, to which the three detection markers C1-C3 are attached. Camera image M2 is an image of the object captured in camera image M1, taken from a different angle.

[0022] The camera orientation for capturing camera images M1 and M2 with respect to the arm tip, which is the target of the image capture, is as shown in Figure 5, for example. The user specifies the position to be the control center for controlling the movement of the robot 10 in each of camera images M1 and M2 (step S2). The user can specify a position on the image by tapping on a touch panel or by operating another pointing device. Here, as shown in Figure 5, the user specifies the center position of the multiple gripping claws of the hand 5 as the position to be the control center. In this case, the user specifies the center position of the gripping claws of the hand 5 in each of camera images M1 and M2. The image processing unit 22 receives such specification operations via the teaching operation device 30. The positions specified by the user on camera images M1 and M2 as the center of the tips of the multiple gripping claws of the hand 5 are designated position P1 and designated position P2, respectively.

[0023] Next, the image processing unit 22 uses a feature extraction model 24 (detection markers, or a 3D model of the robot 10) to extract features from the image of the robot 10, and through image processing, detects the relative position of the specified position in three-dimensional space to the robot 10 (step S3). Here, we describe an example where there are detection markers C1-C3 at the tip of the arm of the robot 10, as shown in Figure 4. In this case, information regarding the image features and arrangement of the detection markers C1-C3 is used as the feature extraction model 24. The appearance of the three detection markers C1-C3 and which detection markers are visible differ depending on the viewing angle of the robot 10. The image processing unit 22 detects the three detection markers C1-C3 by processing each of the camera images M1 and M2, and also analyzes the appearance of the three detection markers C1-C3 to determine from which direction each of the camera images M1 and M2 was captured when viewed from the flange 11. Furthermore, since multiple detection markers C1-C3 exist on each image, and the intervals between these detection markers are known from the feature extraction model 24, the image processing unit 22 can determine the scale of the image (i.e., the correspondence between the size on the image and the actual dimensions).

[0024] Through the image processing described above, the image processing unit 22 determines that the imaging direction relative to the flange 11 for each of the camera images M1 and M2 is as shown in Figure 5. At this time, the image processing unit 22 also utilizes the current position and orientation information of the robot 10. From a single camera image, it can be determined that the specified position on the image lies on an axis in three-dimensional space. In Figure 5, axis L1 represents the axis on which the position corresponding to the specified position P1 in three-dimensional space may exist, as identified by image processing of camera image M1, and axis L2 represents the axis on which the position corresponding to the specified position P2 in three-dimensional space may exist, as identified by image processing of camera image M2. That is, the specified position P1 lies on axis L1 in three-dimensional space, and the specified position P2 lies on axis L2 in three-dimensional space. In this case, the intersection point 101 of axis L1 and axis L2 is the position in three-dimensional space of the control center specified by the user. The image processing unit 22 identifies the position of this intersection point 101. Based on the above, the position in 3D space corresponding to the specified position on the image (relative position with respect to robot 10) is determined (step S3).

[0025] As shown in Figure 6, a flange coordinate system 111 with its origin at the center of the flange surface is pre-set for the robot 10. In step S4, the coordinate system determination unit 23 sets the tool coordinate system 112 by translating the origin of the known flange coordinate system 111 to the control center (intersection 101) identified in step S3.

[0026] Next, we will explain how to determine the relative position between a specified position on an image and the robot 10 by pattern matching using a 3D model of the robot 10 as the feature extraction model 24. In this case as well, similar to when using detection markers, the user is asked to take two images from different directions that include a part of the robot 10 (step S1). Here, we assume that one of the images taken is a camera image M3 that includes the two links on the arm tip side and the hand 5 as shown in Figure 7. In this case, the other image will be an image of the object to be captured in Figure 7 taken from a different angle.

[0027] The user specifies the position they want to be the control center on each image (step S2). Next, the image processing unit 22 determines the relative position between the specified position in each image and the robot 10 (step S3). This will be explained in detail. If the captured image is as shown in Figure 7, the image processing unit 22 performs image detection using the model of the arm tip (hereinafter referred to as the robot model) from the 3D model of the robot 10. For example, the image processing unit 22 may use information on the shape and arrangement of the flat, convex, and concave parts of the arm tip to apply facial recognition technology to identify the posture of the arm tip shown in the image, and then determine the image shooting direction based on the current position and posture information of the robot 10. Alternatively, the image processing unit 22 may extract contours from the image using an edge detection method, perform pattern matching with the contours of the robot model (contours of the arm tip), identify the posture of the arm tip shown in the image, and then determine the image shooting direction based on the current position and posture information of the robot 10.

[0028] Then, the position of the specified location in three-dimensional space is determined as the intersection of the axes of the shooting direction obtained from the two images (i.e., the relative position between the specified location and the robot 10 is determined) (step S3). Next, the coordinate system determination unit 23 sets the tool coordinate system by translating the flange coordinate system to the identified control center (intersection) (step S4).

[0029] According to the control center and coordinate system setting process described above, the user can complete the setting of the control center and coordinate system simply by taking a picture of the robot 10 with the camera 34 of the teaching operation device 30 and specifying the position to be used as the control center on the image. In other words, the operation of setting the control center and coordinate system can be simplified. As for the camera 34, there is no need to perform complex operations such as calibration.

[0030] Alternatively, by capturing three or more images, the control center can be determined from the intersection of the axes of the shooting direction obtained from the three or more images using the same method as described above.

[0031] The above examples illustrate how to identify the control center from two or more images, but it is also possible to identify the control center from a single image. We will now describe an example of identifying the position of the control center in three-dimensional space (relative position to the robot 10) from a single image captured by the user. The procedure is as follows. (A1) The user performs imaging under the restriction that the tip of the robot arm 10 is imaged from directly to the side (along the Y-axis direction of the flange coordinate system). (A2) Under these constraints, the user specifies the position to be used as the control center on the captured image. In this case, as shown in Figure 6, an image parallel to the XZ plane of the flange coordinate system 111 will be captured. (A3) The image processing unit 22 recognizes the scale of the image from the interval between the detected markers displayed in the image, and recognizes the specified position on the image as the X,Z coordinate position, with the flange center (origin of the flange coordinate system) in the image as the reference point. (A4) This identifies the position in 3D space corresponding to the specified position on the image as the X,Z coordinate position in the flange coordinate system. Alternatively, the following method may be used to identify the control center from a single image. In this case, it is assumed that the coordinate system of the controlled object is set on the central axis of the flange coordinate system (i.e., on the Z-axis). The user takes an image including the flange 11. Operationally, it is preferable to take the image from directly beside the object as in the procedure (A1) above, but the direction of acquisition does not need to be particularly restricted. The image processing unit 22 accepts an operation to shift the flange coordinate system in the Z-axis direction (e.g., touch and slide operation) on the image including the flange 11 portion of the robot 10. In this case, the image processing unit 22 may superimpose an image representing the flange coordinate system on the image. The image processing unit 22 identifies the amount of shift of the flange coordinate system on the Z-axis due to the user operation.

[0032] The coordinate system determination unit 23 translates the flange coordinate system to the position of the control center identified in this way and sets the tool coordinate system.

[0033] Even when using a single image, the user can complete the setup of the control center and coordinate system simply by specifying the desired position on the image. Regarding the camera 34, there is no need to perform complex tasks such as calibration.

[0034] The above describes an example of a case where the user specifies the position to be used as the control center on the captured image. The specification of the control center position on the captured image may also be performed automatically by the robot control device 20 (image processing unit 22). For example, methods such as (1) specifying a predetermined position on the captured image (e.g., the center position), or (2) equipping the teaching operation device 30 with a sensor (accelerometer, gyroscope, etc.) that detects the position of the camera 34 in three-dimensional space, and setting the starting point (specified positions P1, P2) when setting the axes L1 and L2 in Figure 5 as the position detected by the sensor (in this case, the teaching operation device 30 provides the robot control device 20 with the captured image and the position information of the camera 34 at the time the image was taken) may be adopted.

[0035] Next, as a specific example of setting the control center as described above, cylindrical fitting will be explained. Cylindrical fitting is the process of inserting a cylindrical workpiece W1, gripped by the hand 5, into a hole formed in the workpiece W2, as shown in Figure 8. Here, we assume that when the workpiece W1 is inserted into the hole, there is sufficient space between the outer surface of the workpiece W1 and the inner surface of the hole, and that force control for insertion is not necessary. In this case, the user sets the control center at the center of the tip surface of the workpiece W1 and sets a tool coordinate system with this control center as the origin. The robot control device 20 controls the robot 10 to move the workpiece W1 until the origin of the tool coordinate system (control center) moves along the central axis of the hole and reaches the bottom of the hole.

[0036] If there are cylindrical workpieces of different lengths to be gripped by Hand 5, the user sets a control center (tool coordinate system) for each length of workpiece. Figure 9 shows the situation where control centers 201, 202, and 203 are set for three different types of workpieces W11, W12, and W13, respectively. For each of the control centers 201-203, a tool coordinate system is set with the vertically downward direction as the Z-axis (see force control coordinate system 113 in Figure 12 for reference).

[0037] Next, we will explain an example of setting parameters related to force control by specifying them on an image, in relation to work performed while force control is being executed. When the gap between the outer surface of the mating workpiece and the inner surface of the hole is narrow when the mating workpiece is inserted into the hole (for example, when the gap is several tens of microns), the mating workpiece will be inserted while constantly in contact with the inner surface of the hole. In this case, the insertion work will be performed by adjusting the posture of the robot 10 while observing the force acting on the mating workpiece. In this case, as shown in Figure 10, the force sensor 3 is positioned between the hand 5 and the flange 11. The force sensor 3 is, for example, a 6-axis force sensor that outputs the load on 3 axes (XYZ) and the moment around each axis as detected values. The force sensor 3 detects the force and moment acting on the mating workpiece W21 during the mating operation. The robot control device 20 (motion control unit 21 and force control unit 25) observes the output of the force sensor 3 and controls the position and posture of the robot 10 so that the moment with respect to the control center set at the center of the tip surface of the mating workpiece W21 becomes zero. When performing force control, the force control unit 25 uses the control center set as described above as the control point for force control (a reference point for calculating moments, etc.). The coordinate system determination unit 23 translates the flange coordinate system to this control point and sets it as the force control coordinate system (see force control coordinate system 113 in Figure 12).

[0038] The image processing unit 22 may be configured to accept the setting of the pressing direction as a parameter related to force control via an image. Two examples of setting the pressing direction as a force control parameter via an image will be explained with reference to Figures 11 and 12.

[0039] As shown in Figure 11, image M4 displays the arm tip of the robot 10, force sensor 3, hand 5, fitted workpiece W21, workpiece W22, and user-defined control center 204. In such image M4, the image processing unit 22 displays an arrow 71 as an image, for example, along the Z-axis direction (vertically downward), to assist the user in setting the pressing direction for force control. If arrow 71 matches the desired pressing direction, the user can select it by, for example, tapping the image of arrow 71. The image processing unit 22 accepts such selection operations from the user. The image processing unit 22 may also display images of three arrows in the X, Y, and Z directions on image M4 and allow the user to select the desired pressing direction.

[0040] As shown in Figure 12, image M5 displays the arm tip of the robot 10, force sensor 3, hand 5, mated workpiece W21, workpiece W22, and user-defined control center 204. The image processing unit 22 accepts user input on image M5, such as specifying the pressing direction with a flick gesture or drawing an arrow (arrow 72 in Figure 12) freehand on the image to indicate the pressing direction. The image processing unit 22 then selects the direction of the axis closest to the direction of the flick gesture or the arrow 72 drawn on the image from among the axes of the force control coordinate system 113 set at the tip of the mated workpiece W21 as the pressing direction. In the example in Figure 12, the Z-axis direction is selected as the pressing direction.

[0041] The force control unit 25 performs force control with respect to the pressing direction specified as described above.

[0042] Next, as another example of specifying force control parameters, an example of setting the range in the depth direction (in this case, the Z-axis direction) where force control can be performed will be explained with reference to Figure 13. Figure 13 is a diagram illustrating an example in which the image processing unit 22 sets the range in the depth direction (range in the direction of tool movement) where force control can be performed as a parameter related to force control on an image. The image M6 shown in Figure 13 displays the arm tip side of the robot 10, force sensor 3, hand 5, fitted workpiece W21, workpiece W22, and the control center 204 set by the user. On such an image M6, the image processing unit 22 accepts an operation by the user to set the range in the depth direction (Z-axis direction) where operation by force control is possible, for example, by drawing a rectangular frame image (rectangular frame 73 in Figure 13) freehand. The force control unit 25 of the robot control device 20 applies force control when the control center (control point) 204 is within the range of the rectangular frame 73 in the Z-axis direction. Note that specifying this rectangular frame can also be expressed as specifying the area in which force control can be performed.

[0043] Next, we will describe an example of setting the tool coordinate system for a hand that is an integrated unit of multiple chuck hands, as shown in Figure 14. The hand 150 shown in Figure 14 is equipped with multiple chuck hands 151-153. Chuck hands 151, 152, and 153 grip workpieces W31, W32, and W33, respectively. It is assumed that control centers 211, 212, and 213 have been set for workpieces W31-W33, respectively, using the method described above. The tool coordinate system is set using the following procedure. Here, a 3D camera capable of detecting the three-dimensional position of the object is used as the camera 34.

[0044] (B1) The image processing unit 22 detects a plane as the leading edge surface of each of the workpieces W31, W32, and W33 from the image including the hand 150. As a method for detecting planes by image processing, for example, a plane detection method using a 3D Hough transform can be used. This detects the leading edge surfaces W31a, W32a, and W33a of the workpieces W31, W32, and W33, respectively. (B2) Next, the coordinate system determination unit 23 determines the rotation matrix in the normal direction of each end face W31a, W32a, and W33a with respect to the Z direction of the flange coordinate system. Then, the coordinate system determination unit 23 translates the origin of the flange coordinate system to the respective control centers 211, 212, and 213, and multiplies it by the rotation matrix determined for each end face, thereby setting the tool coordinate system for each control center 211, 212, and 213.

[0045] Furthermore, in situations where the shooting direction and scale are known from the image of the hand 150, the coordinate system determination unit 23 may set the origin of the tool coordinate system at a point (e.g., the center) on the workpiece tip surface detected by (B1) above, and automatically set the tool coordinate system so that the normal direction of the workpiece tip surface coincides with the Z-axis direction of the tool coordinate system.

[0046] As described above, this embodiment makes it possible to simplify the setting of the control center for controlling the robot's movements.

[0047] Although the present invention has been described above using typical embodiments, those skilled in the art will understand that modifications to the above embodiments and various other modifications, omissions, and additions can be made without departing from the scope of the present invention.

[0048] In the embodiment described above, the distribution of functions within the robot system shown in the block diagram of Figure 2 is just one example, and various modifications are possible. For example, a configuration in which the functions of the image processing unit 22 are placed on the teaching operation device 30 side is also possible.

[0049] In the functional block diagram of the robot control device 20 shown in Figure 2, each functional block may be realized by the CPU of the robot control device 20 executing various software stored in the memory, or by an ASIC (Application Specific Integrated Circuit). This may also be achieved through a hardware-based configuration such as the above.

[0050] The program that performs the control center and coordinate system setting process in the above-described embodiment can be recorded on various computer-readable recording media (for example, semiconductor memory such as ROM, EEPROM, and flash memory, magnetic recording media, optical discs such as CD-ROM and DVD-ROM). [Explanation of symbols]

[0051] 3 Force Sensors 5, 150 hands 10 Robot Manipulators 11 Flange 20 Robot control devices 21 Operation Control Unit 22 Image Processing Unit 23 Coordinate System Determination Unit 24 Feature Extraction Models 25 Force Control Unit 30 Teaching and Operation Device 31 Control Unit 32 Display section 33 Operation section 34 Cameras 100 Robot Systems C1, C2, C3 detection markers

Claims

1. A robot control device for controlling a robotic manipulator equipped with an end effector, An image processing unit that uses a feature extraction model for image detection of the robot manipulator and position and orientation information of the robot manipulator to detect a position in three-dimensional space corresponding to a specified position on the image, as a relative position to the robot manipulator, from an image showing at least a part of the robot manipulator. The system includes a coordinate system determination unit that sets the control center for controlling the operation of the robot manipulator to the position detected in the three-dimensional space by moving the flange coordinate system position, which is set at the flange center of the robot manipulator in a preset location, to the position detected in the three-dimensional space, The image processing unit is a robot control device that uses two or more images, each of which is an image showing at least a part of the robot manipulator and is captured from different directions, to detect a position in the three-dimensional space corresponding to a specified position in each of the two or more images.

2. The robot control device according to claim 1, wherein the image processing unit accepts the user's operation to specify the designated position on the image.

3. The robot control device according to claim 1, wherein the image processing unit automatically sets the designated position at a predetermined position within the image.

4. It further comprises a force control unit for performing force control, The robot control device according to any one of claims 1 to 3, wherein the force control unit sets the control center as a control point for performing the force control.

5. The robot control device according to claim 4, wherein the image processing unit accepts an operation to specify the pressing direction for force control on the image.

6. A robot control device for controlling a robotic manipulator equipped with an end effector, An image processing unit that uses a feature extraction model for image detection of the robot manipulator and position and orientation information of the robot manipulator to detect a position in three-dimensional space corresponding to a specified position on the image, as a relative position to the robot manipulator, from an image showing at least a part of the robot manipulator. A coordinate system determination unit sets the control center for controlling the movement of the robot manipulator to the position detected in the three-dimensional space, It comprises a force control unit for performing force control, The force control unit sets the control center as the control point for performing the force control, The image processing unit is a robot control device that accepts an operation to specify an area on the image in which the force control can be performed.

7. The robot control device according to any one of claims 1 to 6, wherein the coordinate system determination unit sets a coordinate system with the control center as the origin.

8. The aforementioned image shows a workpiece held in the hand acting as the end effector. The image processing unit detects the flat surface of the tip of the workpiece from the image, The robot control device according to claim 7, wherein the coordinate system determination unit sets the coordinate system with a point on the plane as the origin such that the normal direction of the plane coincides with a predetermined axis of the coordinate system.

9. A robotic manipulator equipped with an end effector, A robot control device that controls the robot manipulator, A teaching operation device connected to the robot control device, comprising a teaching operation device equipped with an imaging device, A storage unit for storing a feature extraction model for image detection of the robot manipulator, An image processing unit that uses the feature extraction model and the position and orientation information of the robot manipulator to detect, as a relative position to the robot manipulator, the position in three-dimensional space corresponding to a specified position on the image, from an image captured by the imaging device that shows at least a part of the robot manipulator. The system includes a coordinate system determination unit that sets the control center for controlling the operation of the robot manipulator to the position detected in the three-dimensional space by moving the flange coordinate system position, which is set at the flange center of the robot manipulator in a preset location, to the position detected in the three-dimensional space, The image processing unit detects the position in the three-dimensional space corresponding to a specified position specified in each of the two or more images, using two or more images of at least a part of the robot manipulator, each of which is captured from a different direction.

10. The aforementioned robot manipulator is equipped with a force sensor. The robot system further comprises a force control unit that performs force control based on the detected value of the force sensor, The robot system according to claim 9, wherein the force control unit sets the control center as a control point for performing the force control.

11. The robot system according to claim 9 or 10, wherein the coordinate system determination unit sets a coordinate system with the control center as the origin.

Images