control device

Abstract

A control device (40) for controlling an industrial machine, the control device (40) comprising: an image operating unit (112) that displays an image captured by a visual sensor on a display screen, and accepts a user operation with respect to the image; and a motion control unit (151) that controls a motion of the industrial machine so as to move a visual field of the visual sensor displayed as an image on the display screen, in accordance with the accepted user operation.

Description

[Technical Field]

[0001] The present invention relates to a control device for industrial machinery. [Background technology]

[0002] There is known a robot system that detects a workpiece using a visual sensor attached to the robot's hand and performs a predetermined task such as picking up the detected workpiece. When constructing such a robot system, the robot is taught using a teaching operation panel.

[0003] Examples of robotic systems that use visual sensor functions are described in US Pat. Nos. 5,699,251, 5,999,4 ... and 6,099,520.

[0004] Regarding the robot control system, Patent Document 5 states that "the robot status notification unit 323 notifies the control server 40 of information relating to the robot, for example, when multiple service performing robots 30 are being used, the ID number assigned to each service performing robot 30, the type of robot, the current location of the service performing robot 30, the current status of the service performing robot 30, such as whether it is in standby mode, moving in response to a robot dispatch instruction, or being used by a user, and information on the remaining battery level" (paragraph 0040).

[0005] Patent document 6, regarding a robot teaching system, states that "during a jog feed operation, the operator can switch from an operation mode in which the movement range of the robot 100 is not limited to a predetermined range (free jog feed mode) to an operation mode in which the movement range of the robot 100 is limited to a predetermined range (limited jog feed mode)" (paragraph 0042).

[0006] Patent document 7 states that "a robot control device 2 that controls a robot 1 is provided with a teaching operation panel approach detection device 7 installed outside the operating range of the robot 1, a detectable device 8 that is located on the teaching operation panel 3 and can be detected by the teaching operation panel approach detection device 7, and an operation unit 31 on the teaching operation panel 3, and the robot's operating mode can be selected from the teaching operation panel 3 only when the teaching operation panel approach device 7 detects the teaching operation panel 3" (abstract). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-351570 [Patent Document 2] Japanese Patent Application Publication No. 2020-078859 [Patent Document 3] Japanese Patent Application Laid-Open No. 2005-201824 [Patent Document 4] Japanese Patent Application Laid-Open No. 2006-139525 [Patent Document 5] Japanese Patent Application Publication No. 2017-221991 [Patent Document 6] Japanese Patent Application Laid-Open No. 2008-221428 [Patent Document 7] Japanese Patent Application Laid-Open No. 2004-017223 Summary of the Invention [Problem to be solved by the invention]

[0008] In robot systems that use visual sensors, operators often jog the robot while watching the image captured by the visual sensor when teaching the robot, bringing the workpiece into the visual sensor's field of view. In this case, the orientation of the area containing the workpiece as seen by the operator may differ from the orientation of the workpiece as seen in the image. In such a situation, it is difficult for the operator to intuitively understand how the workpiece will move in the image relative to the direction in which the robot is being jogged. This can result in a lot of effort being spent manipulating the robot to bring the workpiece into the visual sensor's field of view. [Means for solving the problem]

[0009] One aspect of the present disclosure is a control device for controlling industrial machinery, the control device displaying an image captured by a visual sensor on a display screen, and displaying a user operation on the image on the display screen on which the image is displayed. Trace Accepts operation and extracting a trace of the operation of tracing the display screen. and a visual field of the visual sensor that is displayed as the image on the display screen. in a direction according to the locus To move In response to the user operation, and an operation control unit that controls the operation of the industrial machine. [Effects of the Invention]

[0010] According to the above configuration, it becomes possible to operate the robot by operating on the image, and thus to operate the robot intuitively.

[0011] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments of the invention illustrated in the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an overall configuration of a robot system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a robot control device and a teaching pendant. [Figure 3] FIG. 2 is a functional block diagram showing the functional configuration of a teaching pendant, a robot control device, and a visual sensor control device. [Figure 4] 5A and 5B are diagrams illustrating a first embodiment of robot operation based on operations on an image. [Figure 5] 5 is a diagram illustrating a first embodiment of robot operation based on operations on an image, together with FIG. 4. FIG. [Figure 6] FIG. 10 is a diagram illustrating a second embodiment of robot operation based on an operation on an image. [Figure 7] 8A and 8B are diagrams illustrating a third embodiment of robot operation based on operations on an image. [Figure 8] 7A and 7B are diagrams illustrating a third embodiment of robot operation based on operations on an image. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a robot system in which a visual sensor is used as a fixed camera fixed within a workspace. DETAILED DESCRIPTION OF THE INVENTION

[0013] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, like components or functional parts are designated by like reference numerals. The scales of these drawings have been changed appropriately to facilitate understanding. Furthermore, the embodiment shown in the drawings is one example for implementing the present invention, and the present invention is not limited to the illustrated embodiment.

[0014] FIG. 1 is a diagram showing the overall configuration of a robot system 100 according to one embodiment. The robot system 100 includes a robot 30 equipped with a hand (gripping device) 33 at the end of its arm, a robot controller 50 for controlling the robot 30, a teaching operation panel 10 connected to the robot controller 50, a visual sensor 70 attached to the end of the arm of the robot 30, and a visual sensor controller 20 for controlling the visual sensor 70. The visual sensor controller 20 is connected to the robot controller 50. The robot system 100 detects an object (hereinafter referred to as a workpiece) 1 on a workbench 2 using the visual sensor 70, and can handle the workpiece 1 with a hand 33 mounted on the robot 30. Note that FIG. 1 also shows a field of view 7A of the visual sensor 70 that captures an image of the workpiece 1.

[0015] Here, the robot 30 as industrial machinery is assumed to be a vertical articulated robot, but other types of robots may be used. The robot controller 50 controls the operation of the robot 30 in accordance with an operation program loaded into the robot controller 50 or commands input from the teaching pendant 10. As shown in Fig. 1, the entire functions included in the robot controller 50 and the teaching pendant 10 may be defined as a controller 40 of industrial machinery.

[0016] The visual sensor control device 20 has the function of controlling the visual sensor 70 and the function of performing image processing on the image captured by the visual sensor 70. The visual sensor control device 20 detects the position of the workpiece 1 from the image captured by the visual sensor 70 and provides the detected position of the workpiece 1 to the robot control device 50. This allows the robot control device 50 to correct the taught position and perform operations such as removing the workpiece 1. The visual sensor 70 may be a camera that captures grayscale or color images, or a stereo camera or 3D sensor that can acquire distance images or 3D point clouds. The visual sensor control device 20 stores a model pattern of the workpiece and can perform image processing to detect the workpiece by pattern matching the image of the workpiece in the captured image with the model pattern.

[0017] In this embodiment, the visual sensor 70 is assumed to have been calibrated, and the visual sensor control device 20 is assumed to have calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 30. This makes it possible to convert positions on an image captured by the visual sensor 70 into positions on a coordinate system (such as a robot coordinate system) fixed to the workspace.

[0018] In FIG. 1, the visual sensor control device 20 is configured as a device separate from the robot control device 50, but the function of the visual sensor control device 20 may be incorporated into the robot control device 50.

[0019] FIG. 2 is a diagram showing an example of the hardware configuration of the robot control device 50 and the teaching pendant 10. The robot control device 50 may have a configuration similar to that of a general computer, in which a processor 51 is connected to a memory 52 (e.g., ROM, RAM, nonvolatile memory), an input / output interface 53, an operation unit 54 including various operation switches, and the like via a bus. The teaching pendant 10 is used as a device for inputting operations and displaying screens to teach the robot 30 (i.e., to create a control program). The teaching pendant 10 may have a configuration similar to that of a general computer, in which a processor 11 is connected to a memory 12 (e.g., ROM, RAM, nonvolatile memory), a display unit 13, an operation unit 14 including input devices such as a keyboard and a touch panel (software keys), an input / output interface 15, and the like via a bus. Note that various information processing devices such as a tablet terminal, a smartphone, or a personal computer may be used as the teaching device instead of the teaching pendant 10.

[0020] The visual sensor control device 20 may also have a configuration as a general computer, in which a processor is connected to memory (ROM, RAM, non-volatile memory, etc.), an input / output interface, a display unit, an operation unit, etc. via a bus.

[0021] FIG. 3 is a functional block diagram showing the functional configuration of the teaching pendant 10, the robot control device 50, and the visual sensor control device 20.

[0022] 3, the teaching pendant 10 has a teaching unit 111 and an image operation unit 112. The teaching unit 111 presents various information for teaching the robot 30 and functions as an interface that accepts teaching inputs via key operations such as jog operations. The image operation unit 112 displays an image captured by the visual sensor 70 obtained via the robot control device 50 on the display unit 13 and functions as a user interface that accepts user operations on the image (i.e., the display screen configured as a touch panel).

[0023] The robot control device 50 has an operation control unit 151 that controls the operation of the robot 30 in accordance with an operation program or commands from a teaching operation panel. The operation control unit 151 has a movement path calculation unit 152 that calculates a movement path for moving the robot 30 in accordance with a user operation on an image input via the image operation unit 112.

[0024] The teaching pendant 10 has a normal operation mode in which the robot 30 is operated (jog operation, etc.) based on the direction as seen from the robot 30, as well as an operation mode (hereinafter also referred to as an image operation mode) in which the robot 30 is operated by operating on an image. In the image operation mode, the movement path calculation unit 152 calculates a movement path for moving the robot 30 based on information indicating a user operation input via the image operation unit 112. The operation control unit 151 moves the robot 30 (TCP (tool center point)) according to the calculated movement path.

[0025] Switching between the normal operation mode and the image operation mode may be performed by the operator performing a predetermined operation on the operation unit 14. Specifically, a switch button for switching between the normal operation mode and the image operation mode may be provided on the image. Alternatively, switching from the normal operation mode to the image operation mode may be performed automatically in conjunction with launching an adjustment screen for adjusting parameters of the processing program of the visual sensor 70.

[0026] The visual sensor control device 20 has a control unit 121, a storage unit 122, and an image processing unit 123. The control unit 121 controls the visual sensor 70 and the image processing unit 123 and storage unit 122 in an integrated manner in response to commands from the operation control unit 151. The storage unit 122 stores various data (model patterns, etc.) required for image processing, as well as calibration data. The calibration data includes the relative positional relationship between the coordinate system set for the robot 30 and the coordinate system set for the visual sensor 70. The calibration data may also include internal parameters related to the imaging optical system (focal length, image size, lens distortion, etc.). The image processing unit 123 is responsible for performing pattern matching and other various image processing functions.

[0027] Consider a situation in which a robot 30 is being taught to handle a workpiece 1. In this case, an operator generally operates (jogs) the teaching console 10 to move the robot 30 (hand 33) to a desired position while checking the image displayed on the display screen of the teaching console 10 and visually confirming the position. Here, the image displayed on the display screen of the teaching console 10 is an image based on the robot 30 (i.e., an image acquired from the robot's line of sight). Therefore, the orientation of the workpiece displayed on the display screen may differ from the orientation of the workpiece as seen by the operator. In this case, it is difficult for the operator to intuitively grasp how the workpiece will move within the display screen relative to the direction of operation (jogs) of the robot 30. In this embodiment, a configuration is adopted in which a user's operation on the image is accepted, and the robot is controlled so that the area displayed in the visual field of the visual sensor moves in response to the user's operation, thereby enabling intuitive robot operation via an image.

[0028] 4 to 8, examples of robot control by image manipulation will be described below. In these examples, the visual sensor 70 is a camera that captures two-dimensional images, and the images captured by the visual sensor 70 are two-dimensional images. Here, it is assumed that the teaching pendant 10 and the robot control device 50 are set to an image manipulation mode.

[0029] A first embodiment of robot operation by operating on an image will be described with reference to Figures 4 and 5. In Figure 4, the right side shows a state in which an image 13A captured by a visual sensor 70 is displayed on the display screen of the display unit 13 of the teaching pendant 10. As described above, image 13A is an image based on the direction as seen from the robot 30 (an image from the robot's perspective). Image 13A shows the workpiece 1 in the orientation shown. The left side of Figure 4 illustrates the orientation of the workpiece 1 as seen in the operator's field of view E. In this way, the orientation of the workpiece 1 as seen in the image differs from the orientation of the workpiece 1 as seen visually by the operator.

[0030] On this image 13A, the operator can perform an operation (pan operation) to move the field of view of the visual sensor 70. Here, an example is shown in which the operator performs an operation to specify the direction and amount of movement of the field of view of the visual sensor 70 by tracing the image 13A (display screen) with his / her finger. The image operation unit 112 extracts the trajectory of the operator tracing the display screen of the image 13A with his / her finger, and obtains a movement vector T that represents the direction and amount of movement intended by the operator. In the illustrated example, the operator intends to bring the work 1 to the center of the display screen.

[0031] To obtain the movement vector T, the image operation unit 112 may obtain the movement direction by approximating the trajectory of the operator tracing the display screen with a straight line. The image operation unit 112 provides the information representing the user operation thus obtained (here, the movement vector T) to the movement path calculation unit 152 of the robot control device 50.

[0032] The movement path calculation unit 152 calculates the movement path (three-dimensional path information) of the robot 30 from the movement vector T specified on the image. As an example, the movement path calculation unit 152 may project the start point and end point of the movement vector T in the image 13A (the imaging surface of the imaging element) onto a specific plane within the workspace including the robot 30 to obtain a vector within the workspace corresponding to the movement vector T on the image, and set the movement path of the robot 30 from the vector. Note that the operation direction (direction of the movement vector T) by the user's operation on the image is generally opposite to the direction in which the robot 30 is moved. When performing such projection, calibration data including a definition of the relative positional relationship between the visual sensor 70 and the robot 30 is used. The specific plane is, for example, a plane perpendicular to the optical axis of the visual sensor 70. In this example, the specific plane may be the surface on which the workpiece 1 is placed.

[0033] The movement path calculation unit 152 provides the movement path thus obtained to the operation control unit 151. Then, the operation control unit 151 moves the robot 30 according to this movement path. By the above operation, the area displayed on the display screen of the teaching pendant 10 moves according to the user operation (movement vector T). By the above operation, in this example, an image 13B is obtained in which the workpiece 1 is displayed in the center of the display screen, as shown in FIG. 5.

[0034] In the above embodiment, an example of calculating a movement vector representing a user operation was described. However, as long as the movement direction corresponding to the user operation is at least obtained, it is possible to move the visual sensor's field of view in the direction intended by the user. In this case, the image operation unit 112 linearly approximates the trajectory of the operator's finger tracing on the image to obtain the intended movement direction of the operator on the image. This allows the movement path calculation unit 152 to obtain the direction in which to move the robot 30 corresponding to the movement direction on the image. In this case, too, it is possible to obtain the robot's movement direction in the workspace corresponding to the movement direction on the image by using an operation that projects a position on the image representing the user operation onto a specific surface in the workspace. The amount of movement of the robot 30 may be determined appropriately so that the movement amount of the visual sensor 70 is a predetermined amount.

[0035] Furthermore, while the operator is touching and tracing the display screen, the field of view of the visual sensor 70 may be controlled to move in accordance with the tracing operation. As an example, this operation can be realized by the following algorithm. (a1) The start and end points of an operation on an image (panning operation) are acquired at regular time intervals. (a2) The acquired start and end points are projected onto a specific plane in the robot coordinate system. (a3) The robot 30 is controlled by setting a path for the robot to move based on a vector connecting the start point and end point projected onto a specific surface.

[0036] Next, a second example of robot operation by operating on an image will be described with reference to FIG. 6. In this example, the image operation unit 112 displays four arrow keys 201a, 201b, 201c, and 201d indicating up, down, left, and right directions on the display screen of the display unit 13 of the teaching pendant 10 as operation keys for moving the field of view of the visual sensor 70. When the operator touches any of the arrow keys 201a, 201b, 201c, and 201d, the field of view of the visual sensor 70 (the area displayed on the image) moves in the direction indicated by the touched arrow. In this case, the image operation unit 112 provides information indicating the direction of the touched arrow key to the movement path calculation unit 152. The movement path calculation unit 152 calculates the movement direction of the robot 30 within the workspace corresponding to the direction of the touched arrow key.

[0037] The amount of movement may be controlled so that the field of view (the area displayed on the image) of the visual sensor 70 moves at a constant speed while the operator is touching an arrow key. Alternatively, arrow keys of different sizes may be arranged for each direction of movement, and control may be performed so that the larger the arrow, the greater the amount of movement.

[0038] The operator can move the workpiece 1 displayed on the image 13C to a desired position (for example, the center of the screen) by operating the arrow keys.

[0039] Next, a third embodiment of robot operation by operating on an image will be described with reference to FIGS. 7 and 8. In this embodiment, as shown in FIG. 7, an operator specifies an area on an image 13D that reflects the field of view of a visual sensor 70 as an object to be moved on the screen. While FIG. 7 shows an example in which the area 221 specified by the operator is rectangular, the area specified by the operator may have other shapes (such as a circle). The robot control device 50 moves the robot 30 so that the center (geometric center) of the area 221 is located at a predetermined position on the display screen. In other words, the robot control device 50 moves the robot 30 so that the area 221 moves to a predetermined area on the display screen. Here, it is assumed that the center of the area 221 is moved to the center of the display screen.

[0040] In this case, image operation unit 112 calculates a movement vector from the center (geometric center) of area 221 specified on image 13D to the center of image 13D, and provides this movement vector to movement path calculation unit 152. Movement path calculation unit 152 can calculate the movement path of robot 30 from the movement vector on the image using the projection method described above.

[0041] 8 shows image 13E in a state where the center of area 221 has moved to the center of the display screen by the above operation. In this way, when the operator specifies an area on the image, the specified area is moved to a predetermined position on the display screen.

[0042] In this embodiment, there are various possible ways to specify the rectangular area 221. For example, the operator may trace the display screen to create a rectangle, or the operator may specify two points on the screen so that a rectangle with the two points as diagonal corners appears. Alternatively, the operator may touch a single point on the screen so that a rectangle with the point as its geometric center appears. By specifying a rectangular area, the magnification of the image may be changed so that the area is displayed at its maximum size. In this case, for example, the image operation unit 112 may align the center of the specified rectangular area with the center of the display screen as shown in FIG. 8 by the above-described operation, and then adjust the magnification of the image so that the rectangular area is maximized within the display screen.

[0043] A fourth example of robot operation based on image manipulation will be described. Here, we assume that multiple workpieces are displayed in an image on the teaching console 10, and that these workpieces have been detected by previously teaching their shapes and other information. In this case, the operator touches and specifies the workpiece to be moved to a predetermined position (here, the center) on the display screen. Since each workpiece is detected on the image, the position on the image of the workpiece specified by the operator is known by the teaching console 10 or the robot control device 50. Therefore, the image manipulation unit 112 can obtain a movement vector on the image for moving the center of the workpiece specified by the operator to the center of the display screen. Based on this movement vector, the movement path calculation unit 152 can obtain a movement path for the robot 30 for moving the center of the workpiece specified by the operator to the center of the display screen.

[0044] The movement path calculation unit 152 may control the movement amount of the robot 30 in response to the operation amount for the image by adjusting the gain by which the vector as the movement path of the robot obtained as described above is multiplied.

[0045] If the visual sensor 70 is a three-dimensional sensor capable of acquiring three-dimensional position information of an object, when image information acquired by the visual sensor 70 is displayed as an image on the display screen of the teaching pendant 10, three-dimensional path information for moving the robot 30 can be obtained from operations on the image. For example, if the visual sensor 70 is a distance image camera capable of acquiring distance images, depth information of a position specified on the distance image can be obtained, and this depth information can be used to obtain a path within the workspace corresponding to a trajectory specified on the image. Note that even when a distance image is displayed on the teaching pendant 10 and robot control is realized by user operations on the image, it is possible to control the robot in accordance with user operations as described above in the first to fourth embodiments.

[0046] From the viewpoint of improving the safety of robot control in the image operation mode, the operation control unit 151 of the robot control device 50 may be configured to notify the robot 30 while it is moving. For example, while the robot 30 is moving, the operation control unit 151 may perform at least one of the following: vibrating the teaching operation panel 10; outputting a sound from the teaching operation panel 10 (or another device); and displaying a message on the display screen of the teaching operation panel 10 to alert the user that the robot 30 is moving.

[0047] Furthermore, an allowable movement range of the robot 30 may be set in the robot control device 50 so that the robot 30 does not interfere with objects in the workspace. When the allowable movement range is set, the movement control unit 151 controls the robot 30 to stop, for example, when the robot 30 attempts to deviate from the allowable movement range.

[0048] It is also possible to use a position detection sensor to grasp the position of the teaching pendant 10 and control the robot 30 so that it does not enter within a predetermined radius from the position of the teaching pendant 10. For example, a transmitter that outputs a signal (light, radio wave (beacon)) within a predetermined distance range is provided on the teaching pendant 10, and a receiver that receives the signal is provided on the robot 30. By configuring these transmitters and receivers to be controlled by the robot control device 50, it is possible to control the robot 30 so that it does not enter within a predetermined radius (for example, within 1 meter) from the teaching pendant 10. Alternatively, as an example of using a position sensor to grasp the position of the teaching pendant 10, it is possible to grasp the position of the teaching pendant 10 by placing a camera in the work space, or to mount position sensors (acceleration sensor and gyro sensor) on the teaching pendant 10.

[0049] 9 shows a configuration example of a robot system 100A in which the visual sensor 70 is used as a fixed camera fixed in the work space. In this example, the visual sensor 70 is placed on the installation floor in the work space with its field of view 7A facing upward. The robot 30 grasps the workpiece 1 with the hand 33 and makes the workpiece 1 visible to the visual sensor 70, thereby performing processing (detection, judgment, etc.) on the workpiece 1.

[0050] In the robot system 100A, the teaching pendant 10 has the same functions as the teaching pendant 10 shown in Fig. 3. The robot control device 50A has functions equivalent to those of the robot control device 50 shown in Fig. 3, and also has the function of the visual sensor control device 20. In this way, the robot system 100A using the visual sensor 70 as a fixed camera can also realize robot operations in response to operations on images in the case of using the robot system 100 described above.

[0051] In the configuration of the robot system 100A, when the image displayed on the teaching pendant 10 is operated, the robot 30 is controlled so that the workpiece 1 held by the hand of the robot 30 moves in a direction parallel to the plane of the image (the imaging surface of the imaging element). Note that the movement direction of the robot 30 relative to the operation direction on the image is opposite when the visual sensor 70 is mounted on the robot 30 as in the robot system 100 and when the visual sensor 70 is fixedly installed as in the robot system 100A. Note that in this example, since the subject of the visual sensor 70 is the workpiece 1, moving the position of the robot 30 relative to the visual sensor 70 so that the workpiece 1 displayed as an image on the display screen moves in response to a user operation on the image is equivalent to moving the field of view of the visual sensor 70 displayed as an image on the display screen in response to a user operation on the image.

[0052] According to the embodiment described above, it is possible to operate a robot by operating on an image, and thus to operate the robot intuitively.

[0053] Although the present invention has been described using exemplary embodiments, those skilled in the art will appreciate that modifications and various other changes, omissions, and additions can be made to the above-described embodiments without departing from the scope of the present invention.

[0054] The various functions realized by the robot control device and teaching pendant in the above-described embodiment can be applied to control devices for various industrial machines, including control devices for machine tools.

[0055] 3 is an example, and various configuration examples are possible regarding the arrangement of the functional blocks. For example, the movement path calculation unit 152 may be incorporated in the teaching pendant.

[0056] The functional blocks realized in the visual sensor control device, robot control device, and teaching operation panel shown in Figure 3 may be realized by the processors of these devices executing various software stored in memory devices, or may be realized by a hardware-based configuration such as an ASIC (Application Specific Integrated Circuit).

[0057] The programs for executing the functions of the visual sensor control device, robot control device, and teaching operation panel in the above-mentioned embodiments can be recorded on various computer-readable recording media (e.g., semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical disks such as CD-ROM and DVD-ROM). [Explanation of symbols]

[0058] 1 Work 2 workbenches 10 Teaching control panel 11 processors 12 Memory 13 Display section 13A-13E images 14 Control section 15 Input / Output Interface 20 Visual sensor control device 30 Robot 33 hands 40 Control device 50, 50A Robot Control Device 51 processors 52 memory 53 Input / Output Interface 54 Control section 70 Visual Sensor 100, 100A Robot System 111 Teaching Department 112 Image control section 121 Control Unit 122 Storage section 123 Image Processing Unit 151 Motion control section 152 Movement path calculation unit

Claims

1. A control device for controlling an industrial machine, an image operation unit that displays an image captured by the visual sensor on a display screen, and receives a user operation on the image by tracing the display screen on which the image is displayed, and extracts a trajectory of the operation by tracing the display screen; an operation control unit that controls the operation of the industrial machine in accordance with the user operation so that the field of view of the visual sensor displayed as the image on the display screen moves in a direction corresponding to the trajectory.

2. The control device according to claim 1 , wherein the operation control unit includes a movement path calculation unit that calculates a movement path of the industrial machine in response to the user operation.

3. The image operation unit determines the direction in which the field of view of the visual sensor moves by linearly approximating the trajectory on the display screen; The control device according to claim 2 , wherein the movement path calculation unit calculates the movement path based on the determined direction.

4. the image operation unit determines a start point and an end point representing the trajectory based on the trajectory on the display screen; The control device according to claim 2 , wherein the movement path calculation unit calculates the movement path based on the determined start point and end point.

5. the image operation unit acquires the start point and the end point at regular time intervals while the user operation is being performed; 5. The control device according to claim 4, wherein the movement path calculation unit determines the movement path based on the start point and the end point acquired at regular time intervals so that the field of view of the visual sensor moves in accordance with a tracing operation on the display screen.

6. The control device according to claim 2 , wherein the movement path calculation unit determines the movement amount of the industrial machine by multiplying the movement amount on the image represented by the user operation by a predetermined gain.

7. the image is a two-dimensional image, The control device according to claim 2 , wherein the movement path calculation unit calculates the movement path by using an operation of projecting a position on the image that represents the user operation onto a predetermined surface in a workspace.

8. The visual sensor is fixed in position within the workspace; the industrial machine uses a gripping device to grip an object to be imaged by the visual sensor; The control device according to claim 1 , wherein the operation control unit moves the industrial machine in response to the user operation so that a position of the object displayed as the image on the display screen moves.

9. The control device according to claim 1 , wherein the operation control unit notifies the user that the industrial machine is moving while the industrial machine is being moved.

10. The control device according to claim 1 , wherein the operation control unit controls the industrial machine so that the industrial machine operates within a preset allowable operation range.

Images