CONTROL DEVICE, ROBOT SYSTEM AND CONTROL METHOD

Abstract

The present invention provides a control device capable of high-speed, high-precision processing. This control device receives an output from a visual sensor that captures an image of a moving workpiece and controls processing performed on the workpiece. The control device includes a two-dimensional image acquisition unit, a three-dimensional information generation unit, a two-dimensional image processing area restriction unit, and a motion control unit. The two-dimensional image acquisition unit acquires a two-dimensional image that includes the workpiece based on the output of the visual sensor, and the three-dimensional information generation unit generates three-dimensional information that includes the workpiece based on the output of the visual sensor.The processing area restriction unit for a two-dimensional image restricts the processing area in the acquired two-dimensional image, and the motion control unit changes a relative position of the three-dimensional information based on the processing area, which is limited by the two-dimensional image in a predetermined coordinate system.

Description

AREA

[0001] The present disclosure relates to a control device, a robot system and a control method. BACKGROUND

[0002] Traditionally, a control device (robot controller) is provided that detects the surface shape of a workpiece (object), its position, and similar information based on an image captured by a visual sensor (imaging device), and instructs the robot to perform a predetermined task on the workpiece. It should be noted that the visual sensor used can be either a two-dimensional sensor that captures a two-dimensional image of the workpiece or a three-dimensional sensor that measures three-dimensional information such as the distance between the visual sensor and the workpiece.

[0003] A conventional robot system, which controls a robot equipped with a work tool (end effector) and performs a predetermined task on a moving workpiece, is also practically implemented using the control device described above. In such a robot system, the control device calculates the position and orientation of the workpiece based on the two-dimensional image and three-dimensional information obtained from the visual sensor output, and controls tasks such as grasping the workpiece by controlling the robot's hand (work tool). It should be noted that the movement (conveying) of the workpiece is carried out, for example, by a conveyor, a conveying tool, or similar device.

[0004] Incidentally, in the past various control techniques have been proposed for controlling a predetermined work on a workpiece based on the two-dimensional image and the three-dimensional information obtained from the output of the visual sensor. [LIST OF QUOTES][PATENT LITERATURE] [PTL 1] International Patent Publication No. 2021-256437 [PTL 2] Unexamined Japanese patent publication (Kokai) No. H05(1993)-173644 SUMMARY [TECHNICAL PROBLEM]

[0005] As mentioned above, robot systems have been used in practice that utilize, for example, two-dimensional images obtained from the output of a two-dimensional sensor and three-dimensional information generated from the output of a three-dimensional sensor to control a robot to perform specified tasks on a moving workpiece.

[0006] However, if the surface of a conveyor transporting a workpiece in such conventional robotic systems is mesh-like, there is a risk of reduced accuracy in capturing the workpiece or its characteristic parts in a two-dimensional image. Furthermore, if the capture processing is based on a two-dimensional image that includes a workpiece and three-dimensional information, the amount of data to be processed increases, requiring a high-performance arithmetic processing device or making it difficult to perform the capture processing at high speed.

[0007] Therefore, there is a demand for a control device, a robot system and a control method that are highly accurate and capable of high-speed processing. [SOLUTION TO THE PROBLEM]

[0008] According to one embodiment of the present disclosure, a control device is provided which is configured to receive an output from a visual sensor which captures an image of a moving workpiece and to control the processing of the workpiece, comprising a two-dimensional image acquisition unit, a three-dimensional information generation unit, a two-dimensional image processing area restriction unit and a motion control unit.

[0009] The two-dimensional image acquisition unit acquires a two-dimensional image that includes the workpiece, based on the output of the visual sensor, and the three-dimensional information generation unit generates three-dimensional information that includes the workpiece, also based on the output of the visual sensor. The two-dimensional image processing area restriction unit restricts a processing area within an acquired two-dimensional image, and the motion control unit modifies the relative position of the three-dimensional information based on the processing area constrained by the two-dimensional image in a predetermined coordinate system. BRIEF DESCRIPTION OF DRAWINGS [ Fig. 1] Fig. Figure 1 is a front view that schematically depicts an overall configuration of a first embodiment of a robot system according to the present embodiment. [ Fig. 2] Fig. 2 is a top view of the in Fig. 1 robot system shown. [ Fig. 3] Fig. 3 is a functional block diagram of a robot system that is in Fig. 1 is shown. [ Fig. 4] Fig. Figure 4 is a diagram to explain an example of the visual sensor in which in Fig. 3 function block diagrams shown. [ Fig. 5] Fig. Figure 5 is a flowchart to explain a control processing example by the robot controller (control device) in which the following is shown: Fig. 3 function block diagrams shown. [ Fig. 6] Fig. Figure 6 is a diagram explaining how three-dimensional information about the workpiece is generated. [ Fig. 7] Fig. Figure 7 is a diagram to explain a control processing example using the processing area restriction unit for a two-dimensional image, in which Fig. 3 function block diagrams shown. [ Fig. 8] Fig. Figure 8 is a diagram explaining a tax processing example using the two-dimensional image collection unit, in which Fig. 3 function block diagrams shown. [ Fig. 9] Fig. Figure 9 is a diagram to explain another processing example by the processing area restriction unit for a two-dimensional image in which Fig. 3 function block diagrams shown. [ Fig. 10] Fig. Figure 10 is a flowchart to explain another control processing example by the robot controller (control device) in which in Fig. 3 function block diagrams shown. [ Fig. 11] Fig. Figure 11 is a top view schematically depicting an overall configuration of a second embodiment of a robot system according to the present embodiment. DESCRIPTION OF EXECUTION FORMS

[0010] Examples of a control device, a robot system, and a control method according to the present embodiment are described in detail below with reference to the accompanying drawings. The same or similar reference numerals are assigned to the same or similar elements in each drawing. Furthermore, the embodiments described below do not limit the technical scope of the invention as described in the claims or the definitions of the terms.

[0011] Fig. Figure 1 is a front view schematically depicting an overall configuration of a first embodiment of a robot system according to the present embodiment, and Fig. 2 is a top view of the in Fig. 1 robot system shown. In Fig. 1 and Fig. Reference numeral 2 represents a robot system of a first embodiment, 1 is a robot, 2 is a hand, 3 is an imaging device, 6 is a conveyor, and 81 indicates a workpiece (object). As in Fig. 1 and Fig. Figure 2 shows that the robot system 100 comprises the robot 1, which includes the hand 2, the conveyor 6 and the imaging device 3.

[0012] The robot 1 includes a base section 14, which is attached to the installation surface; a rotary base 13, which rotates relative to the base section 14; a lower arm 12, which is rotatably supported by the rotary base 13; an upper arm 11, which is rotatably supported by the lower arm 12; and a wrist 15, which is rotatably supported by an end section of the upper arm 11. It should be noted that the upper arm 11 rotates about an axis of rotation parallel to the direction in which the upper arm 11 extends, and that a rotatably formed flange 16 is provided at the tip of the wrist 15. In the description of the present embodiment, the robot 1 is a multi-joint robot arm having a plurality of joints, but it is not limited to this configuration, and any robot capable of controlling various work tools can be used.Although workpiece 81 is shown as an example of a rectangular, parallelepiped corrugated cardboard box, workpiece 81 is furthermore not limited to corrugated cardboard.

[0013] The hand 2 is a working tool capable of grasping the workpiece 81 and is configured for attracting and gripping the workpiece 81, for example, by means of the multiple suction cups 2a. Although the hand 2 is attached to the flange 16 of the wrist 15, any working tool can be used instead of the one attached to the robot 1.

[0014] The conveyor 6 is an example of a moving device that conveys the workpiece 81 and conveys the workpiece 81 in a predetermined direction by rotating the annular belt 6a. In particular, the conveyor 6 moves the workpiece 81 in the horizontal direction, as indicated by the arrow 86, and conveys the workpiece 81 to a position where the hand 2 can grasp the workpiece 81 by changing the positions and orientation of the robot 1.

[0015] The imaging device 3 includes a two-dimensional sensor (visual sensor 30) for capturing a two-dimensional image of the workpiece 81, and a three-dimensional sensor (visual sensor 30) for acquiring three-dimensional information on the surface of the workpiece 81. In particular, based on the output of the visual sensor 30, the two-dimensional image of the workpiece 81, which is conveyed by the conveyor 6, can be captured, and three-dimensional information (three-dimensional point cloud data, three-dimensional position information) on the surface of the workpiece 81 can be acquired. It should be noted that reference numeral 24 indicates a conveyor drive motor that drives the annular belt 6a of the conveyor 6, and reference numeral 25 indicates, for example, a position detector such as an encoder. An example of the visual sensor 30 is described in detail below with reference to Fig. 4 described.

[0016] The visual sensor 30 is supported by the support element 37 and is positioned so that it can image the workpiece 81, which is conveyed by the conveyor 6. The visual sensor 30 is located upstream of the robot 1 in the direction in which the workpiece 81 is conveyed. It should be noted that the world coordinate system 76 is defined as the reference coordinate system in the robot system 100, and, for example, the origin of the world coordinate system 76 is located on the base section 14 of the robot 1. Consequently, the position and orientation of the world coordinate system 76 do not change even if the position and orientation of the robot 1 change. The world coordinate system 76 has an X-axis, a Y-axis and a Z-axis that are orthogonal to each other as coordinate axes, and a W-axis, a P-axis and an R-axis that are defined as coordinate axes around the X-axis, the Y-axis and the Z-axis.

[0017] Furthermore, a tool coordinate system 77 is defined in the robot system 100, which has an origin that is defined at an arbitrary position of the working tool, and an origin of the tool coordinate system 77 is defined at a tool tip point of hand 2. Consequently, when the robot 1 changes its position and orientation, the position and orientation of the tool coordinate system 77 change; for example, the position of the robot 1 corresponds to the position of the tool tip point, and the orientation of the robot 1 corresponds to the orientation of the tool coordinate system 77 relative to the world coordinate system 76. A sensor coordinate system 78 is defined in the robot system 100, which corresponds to the visual sensor 30, and the origin of the sensor coordinate system 78 is a coordinate system that is attached to the visual sensor 30.Therefore, the coordinate values ​​in the sensor coordinate system 78 can be converted into coordinate values ​​in the world coordinate system 76 based on the position and orientation of the sensor coordinate system 78 in relation to the world coordinate system 76.

[0018] Fig. 3 is a functional block diagram of a robot system that is in Fig. Figure 1 shows that the robot system 100 further includes a robot controller (control device) 4 and a conveyor control device 5. As shown in Fig. 1, Fig. 2 to Fig. Figure 3 shows that the robot 1 includes a plurality of robot drive motors 22, which change the position and orientation of the robot 1, and a position detector 23, such as an encoder, attached to each robot drive motor 22. The hand 2 includes, for example, a pump 21 for reducing the pressure within the suction cup 2a in order to suction the workpiece 81.

[0019] The robot controller 4 includes a memory unit 42, an operating control unit 43, a hand drive unit 44, a robot drive unit 45, a display 46, and an image processing unit 47. It should be noted that the operating control unit 43 and the image processing unit 47 are configured by a microprocessor (MPU), which functions as an arithmetic processing unit and performs various processing operations by exchanging data between RAM (random access memory), ROM (read-only memory), and the memory unit 42, which is a flash memory. An operating program 41 for controlling, for example, a robot 1, the hand 2, and the conveyor 6, is stored in the memory unit 42 and is executed by the arithmetic processing unit (operating control unit 43, image processing unit 47, etc.).

[0020] The operating control unit 43 issues an operating command according to the operating program 41 to the hand drive unit 44 and the robot drive unit 45, and controls the hand 2 and the robot 1. Furthermore, the operating control unit 43 issues an operating command according to the operating program 41 to the image processing unit 47 and controls the acquisition of the two-dimensional image based on the output of the visual sensor 30, the generation of the three-dimensional information, image processing, and the like. The display 46 consists, for example, of a liquid crystal display panel or the like and displays information or the like regarding the control of the hand 2, the robot 1, and the conveyor 6.

[0021] The conveyor control device 5 includes a storage unit 52, an operating control unit 53, and a conveyor drive unit 54. It controls the conveyor 6 based on a command from the robot controller 4 and transports the workpiece 81 to a predetermined position. Note that the conveyor 6 includes a conveyor drive motor 24 and a position detector 25. The position detector 25 can be configured as an encoder (rotary encoder), for example, mounted on an output shaft of the conveyor drive motor 24, and can detect a schematic position of the workpiece 81 being transported by the conveyor 6 based on the encoder output. It should be noted that the position detector 25 is not limited to the encoder and could, for example, be a photoelectric sensor or the like.

[0022] The image processing unit 47 includes a generation unit 61 for three-dimensional information, a acquisition unit 62 for two-dimensional images, a motion control unit 63, a processing area restriction unit 64 for two-dimensional images, a feature section acquisition unit 65, and a computation unit 66. The generation unit 61 for three-dimensional information generates three-dimensional information based on the output of the three-dimensional sensor, and the acquisition unit 62 for two-dimensional images acquires the two-dimensional image based on the output of the two-dimensional sensor.The motion control unit 63 corrects the relative position of the three-dimensional information with respect to the two-dimensional image, and the two-dimensional image processing area restriction unit 64 restricts the two-dimensional image to a narrower area from the two-dimensional image acquired by the two-dimensional image acquisition unit 62 and outputs the restricted two-dimensional image as a processing area.

[0023] The feature section detection unit 65 detects a predetermined feature section of the workpiece 81, and the calculation unit 66 calculates the position and orientation of the robot 1 based on the position and orientation of the workpiece 81. It should be noted that the image processing unit 47 is not integrated into the robot controller 4, but can be provided in a separate body outside the robot controller 4. The conveyor control device 5 can also be integrated into the robot controller 4 and can be modified and deformed in various ways.

[0024] Fig. Figure 4 is a diagram to explain an example of the visual sensor in which in Fig. 3 function block diagrams shown. As in Fig. As shown in Figure 4, the visual sensor 30 (imaging device 3) includes a first camera 31, a second camera 32, and a projector 33. It should be noted that although the first camera 31 and the second camera 32 function as a stereo camera, each camera 31 or 32 also functions as a two-dimensional camera, capturing a two-dimensional image. It should be noted that any camera incorporating an imaging element, such as a charge-coupled device sensor (CCD sensor) or a complementary metal-oxide-semiconductor sensor (CMOS sensor), can be used as the cameras 31 and 32. Furthermore, the two cameras 31 and 32 are positioned at a predetermined distance from each other.

[0025] For example, the projector 33 projects pattern light, such as a stretched pattern, onto the workpiece 81, and the cameras 31 and 32 and the projector 33 are arranged within the housing 34. It should be noted that when the three-dimensional generating unit 61 generates information based on the output of the visual sensor 30, for example, the projector 33 projects the pattern light and uses the stereo image (two two-dimensional images) from the first camera 31 and the second camera 32. When the two-dimensional image acquisition unit 62 acquires the two-dimensional image based on the output of the visual sensor 30, for example, the pattern light from the projector 33 is stopped, and the two-dimensional image captured by either the first camera 31 or the second camera 32 is used.The timing with which the generation unit 61 processes the output of the visual sensor 30 for three-dimensional information is therefore preferably different from the timing with which the acquisition unit 62 acquires the output of the visual sensor 30 for a two-dimensional image. However, if the visual sensor 30 includes, for example, a dedicated three-dimensional sensor for output to the generation unit 61 for three-dimensional image processing, such as a time-of-flight (TOF) method, and a dedicated two-dimensional sensor for output to the acquisition unit 62 for two-dimensional image processing, it is also possible to perform processing without providing a timing difference. Furthermore, it is understood that various other known three-dimensional and two-dimensional sensors besides the examples mentioned above can be used as the visual sensor 30.

[0026] As in Fig. 3 and Fig. As shown in Figure 4, the generation unit 61 for three-dimensional information can generate three-dimensional information about the surface of an imaging target as three-dimensional point group data (three-dimensional map) by processing the image acquired by the visual sensor 30. It should be noted that the three-dimensional information includes, for example, information about the positions of the multitude of measurement points defined on the surface of the imaging target. The three-dimensional point group data represents the position of the surface of the imaging target by a set of coordinate values ​​(x, y, z) of the measurement points defined on the surface of the imaging target.

[0027] The three-dimensional information generation unit 61, for example, defines a multitude of measurement points on the surface of the imaging target within the imaging area 35 of the visual sensor 30. These measurement points can be defined for each pixel of the two-dimensional image from camera 31 or camera 32. Furthermore, the three-dimensional information generation unit 61 can calculate the distance from the visual sensor 30 to each measurement point based on the parallax of the two-dimensional image captured by the two cameras 31 and 32. The three-dimensional information generation unit 61 then calculates the coordinate value of the measurement point in a sensor coordinate system 78 based on the distance from the visual sensor 30 to each measurement point.Alternatively, the coordinate value of the sensor coordinate system 78, based on the position and orientation of the visual sensor 30, can also be converted into the coordinate value of the world coordinate system 76. In this way, the generation unit 61 for three-dimensional information generates three-dimensional point group data (three-dimensional information) that includes the coordinate values ​​of the multitude of measurement points.

[0028] The motion control unit 63 receives the output from the position detector 25 of the conveyor 6 and calculates the actual amount of movement by which the workpiece 81 moved from the time the stereo image was acquired until the time the two-dimensional image was acquired, in order to generate the three-dimensional information. The motion control unit 63 then performs control operations to move the three-dimensional information so that it corresponds to the two-dimensional image in a predetermined coordinate system, thus matching the amount of movement of the workpiece 81. This control allows a portion of the three-dimensional information of the workpiece 81 and the two-dimensional image of the workpiece 81 to be superimposed in the coordinate system described above.In other words, the three-dimensional information and the two-dimensional image that corresponds to the three-dimensional information and the two-dimensional image that is simultaneously acquired can be generated.

[0029] Fig. Figure 5 is a flowchart to explain a control processing example by the robot controller (control device) in which the following is shown: Fig. 3 function block diagrams shown. As in Fig. Figure 5 shows an example of control processing by the robot controller according to the first embodiment (START). In step ST1, a conveyor control device 5 controls a conveyor 6 to move a workpiece 81 in the direction indicated by an arrow 86. Fig. 6 is specified, and the workpiece 81 is enclosed within an imaging area 35 of the visual sensor 30. It should be noted that based on an output from a position detector 25, it can be determined whether the workpiece 81 is located within the imaging area 35 or not.

[0030] The process then proceeds to step ST2, where a three-dimensional information generation unit 61 generates three-dimensional information based on an output from the visual sensor 30 (e.g., the stereo image). At this point, the position detector 25 detects the first rotational position of a conveyor drive motor 24 and outputs this detected rotational position to the conveyor control device 5. Furthermore, in step ST3, a two-dimensional image acquisition unit 62 acquires a two-dimensional image based on the output from the visual sensor 30 (e.g., a two-dimensional image from the first camera 31). At this point, the position detector 25 detects the second rotational position of the conveyor drive motor 24 and outputs this detected rotational position to the conveyor control device 5.

[0031] It should be noted that the conveyor 6 can be equipped with a sensor (for example, a photoelectric sensor) that detects that the workpiece 81 has arrived at the imaging area 35 and can be configured to detect the rotational position (first and second rotational position) of the position detector 25 based on the sensor output. Furthermore, the first and second rotational positions output by the position detector 25 are stored and kept, for example, in a memory unit 52.

[0032] Furthermore, the process continues to step ST4, where the movement distance of the workpiece 81 in the conveyor 6 is calculated, and the process continues to step ST5. In step ST5, a motion control unit 63 corrects the relative position of the three-dimensional information for the two-dimensional image and continues to step ST6. In step ST6, a processing area restriction unit 64 for two-dimensional image calculates (restricts) a processing area from the two-dimensional image acquired by the acquisition unit 62 for two-dimensional image and continues to step ST7. The restriction of the processing area by the processing area restriction unit 64 for two-dimensional image is described below with reference to Fig. 6, Fig. 7, Fig. 8 to Fig. 9 described.

[0033] In step ST7, a feature section capture unit 65 captures a feature section of the workpiece 81 in the two-dimensional image, where the processing area is restricted by the processing area restriction unit 64 for two-dimensional image. It should be noted that, for example, if a top surface of the workpiece 81 is specified as the feature section, the reference image of the top surface of the workpiece 81 is pre-stored in a storage unit 42. The reference image can also be an image in which the top surface of the workpiece 81 is actually captured by the two-dimensional camera, or it can generate a reference image of the workpiece 81 based on the three-dimensional data obtained from the CAD (computer-aided design) device.The feature section acquisition unit 65 captures an image on the upper surface of the workpiece 81 in a two-dimensional image, in which the processing area is restricted by the processing area restriction unit 64 for two-dimensional image by a procedure such as template matching using the reference image.

[0034] As described above, according to the control device (robot system) of the present embodiment, processing can be carried out based on the two-dimensional image even if the surface of the conveyor 6 (belt 6a) has a mesh shape or the like, in which the processing area is restricted by the processing area restriction unit 64 for two-dimensional images. This can, for example, reduce the likelihood of incorrect detection of the workpiece 81. Furthermore, the processing area of ​​the two-dimensional image restricted by the processing area restriction unit 64 for two-dimensional images can speed up processing because the processing area (surface) becomes narrower than the two-dimensional image acquired by the acquisition unit 62 for two-dimensional images.

[0035] In step ST8, the computation unit 66 subsequently calculates the position and orientation of the workpiece 81. It should be noted that the position of the workpiece 81 can be defined as, for example, the center of its centroid, a specific position, or the position of an end section of an upper surface of the workpiece 81, and the orientation of the workpiece can be defined as a normal direction or similar characteristic of the upper surface of the workpiece 81. The computation unit 66 extracts a measurement point located within an area overlapping Figure 72a and calculates the position and orientation of the workpiece 81 based on the coordinate values ​​of the plurality of measurement points.

[0036] Furthermore, in step ST9, the calculation unit 66 calculates a position and orientation of the robot 1 based on the position and orientation of the workpiece 81. Specifically, the calculation unit 66 calculates the position and orientation of the workpiece 81 when the workpiece 81 is moved by the conveyor 6 to a position where the workpiece 81 is grasped by a hand 2, and calculates the position and orientation of the robot 1 based on the position and orientation of the workpiece 81 at that time.

[0037] Then, in step ST10, the computing unit 66 transmits a command to drive the robot 1 and the hand 2 to an operating control unit 43. The operating control unit 43 drives the robot 1 and the hand 2 based on the command from the computing unit 66, the robot 1 (hand 2) grasps and conveys the workpiece 81 and completes (END) an example of control processing by the robot controller according to the first embodiment.

[0038] As described above, according to an example of control processing by the robot controller according to the first embodiment, since the processing area restriction unit 64 for two-dimensional image restricts the processing area of ​​the two-dimensional image acquired by the acquisition unit 62 for two-dimensional image, acquisition processing or the like is performed based on the actually acquired two-dimensional image, so that erroneous acquisition of the workpiece 81 or the like can be reduced. Moreover, since the area of ​​the processing area for two-dimensional image restricted by the processing area restriction unit 64 for two-dimensional image is narrower than the two-dimensional image acquired by the acquisition unit 62 for two-dimensional image, the processing can be accelerated.

[0039] Fig. Figure 6 is a diagram to explain how three-dimensional information (three-dimensional point group data) of the workpiece is generated, and Fig. Figure 7 is a diagram explaining a processing example using the processing area restriction unit for a two-dimensional image in the function block diagram shown in Fig. 3 is shown. First, as in Fig. Figure 6 shows a workpiece 81 being transported by a conveyor 6 in the direction of arrow 86, and an imaging area 35 encompassing the workpiece 81 is captured by the visual sensor 30. Fig. Reference numeral 71 indicates a two-dimensional image (measuring area) acquired by a two-dimensional image acquisition unit 62; reference numeral 71a indicates an image area of ​​the workpiece 81; reference numeral 71b indicates image areas (measuring points PX) of both ends of a belt 6a of the conveyor 6; reference numeral 71c indicates an image area of ​​a transport section of the belt 6a; and reference numeral 71d indicates an image area of ​​a floor surface on which the conveyor 6 is installed. Reference numeral 91 further indicates a processing area restricted by a two-dimensional image processing area restriction unit 64, and references numerals 91a to 91c indicate boundaries.

[0040] As in Fig. Figure 7 shows that the processing area restriction unit 64 for two-dimensional image adds borders 91a to 91c to an image area 71a of the workpiece 81 in the two-dimensional image 71, which is collected by the two-dimensional image acquisition unit 62, to specify a processing area 91 for the actual processing operation. It should be noted that the processing area 91 is defined such that a border 91c in front of the direction in which the workpiece 81 is moving is wider than a border 91a behind the direction in which the workpiece 81 is moving, and the borders 91b and 91c on both sides in the direction in which the workpiece 81 is moving are narrower than the border 91a behind the direction in which the workpiece 81 is moving.

[0041] Since edge 91c is a section before the workpiece 81 is reached, it is assumed that edge 91c is preferably set wider than edge 91a, i.e., the section after the workpiece 81 is reached. Since edge 91b is displaced very little in the direction perpendicular to the direction of movement of the workpiece 81, it is assumed that there is no problem even if edge 91b is set narrower than edge 91a, i.e., the section after the workpiece 81 is reached. It should be noted that the processing area restriction process by the processing area restriction unit 64 for two-dimensional images is not limited to those described above.

[0042] Fig. Figure 8 is a diagram explaining a tax processing example using the two-dimensional image collection unit, in which Fig. 3 function block diagrams shown. In Fig. 8 is a position of the workpiece 81, used to generate the three-dimensional information, indicated by a dashed line. Although the workpiece 81 is moved by the conveyor 6 in a direction indicated by an arrow 86, the first camera 31 of the visual sensor 30, for example, captures the two-dimensional image of the workpiece 81 when the workpiece 81 is conveyed within the imaging area 35. Specifically, the visual sensor 30 can capture the two-dimensional image immediately after imaging to generate the three-dimensional information. Alternatively, the visual sensor 30 can capture the two-dimensional image after a predetermined time has elapsed following imaging to acquire the three-dimensional information.

[0043] As in Fig. As shown in Figure 8, the motion control unit 63 performs motion control for each measuring point. For example, the three-dimensional information includes a coordinate value of a measuring point P1A located on the surface of the workpiece 81. The motion control unit 63 processes the measuring point P1A based on the amount of movement of the workpiece 81 in the direction in which the workpiece 81 is moving, and performs processing such that the measuring points P1A and P2A move to the measuring points P1B and P2B. It should be noted, for example, that in a predetermined coordinate system (e.g., the sensor coordinate system 78, which is shown in Figure 8), the measuring points P1A and P2A move to the measuring points P1B and P2B. Fig. (as shown in Figure 1), the motion control unit 63 changes the relative position of the three-dimensional information generated by the three-dimensional information generation unit 61, based on the processing area of ​​the two-dimensional image restricted by the two-dimensional image processing area restriction unit 64.

[0044] Fig. Figure 9 is a diagram to explain another processing example by the processing area restriction unit for a two-dimensional image in which Fig. 3 shown in the function block diagram. The restriction process of the processing area by the processing area restriction unit 64 for a two-dimensional image, which refers to Fig. 7, restricts the processing area 91, which is provided with borders 91a to 91c around an image area 71a of the workpiece 81, but the restriction process of the processing area by the processing area restriction unit 64 for two-dimensional image, which with reference to Fig. As described in section 9, the processing area is restricted in such a way that it corresponds to the image area of ​​the workpiece 81 in the two-dimensional image acquired by the image acquisition unit 62 for two-dimensional imaging. Fig. Reference code MK, for example, indicates a label, such as a two-dimensional code, affixed to the top surface of workpiece 81 of the corrugated fiberboard box, and reference code 79 indicates an image coordinate system. It should be noted that reference codes 72, 72a, 72b, and 72c indicate sections 71, 71a, 71b, and 71c of the section described above. Fig. 7 corresponds to, and 87 indicates a direction in which the workpiece 81 moves.

[0045] As in Fig. As shown in Figure 9, for example, when reading the marking MK, which is attached to a surface of the workpiece 81, by a visual sensor 30 to perform a predetermined operation, the processing area restriction unit 64 for two-dimensional image restricts the image area 72a (71a) of the workpiece 81 as a processing area for actual processing in the two-dimensional image 72, which is acquired by the acquisition unit 62 for two-dimensional image. Specifically, in the case of reading the marking MK, which is attached to the upper surface of the workpiece 81, it is possible to read the marking MK even if the processing area restricted by the processing area restriction unit 64 for two-dimensional image is the image area of ​​the workpiece 81.

[0046] In particular, by correcting the three-dimensional information in step ST5 of the flowchart described above Fig. Figure 5 illustrates, for example, that the three-dimensional information (three-dimensional position information) is obtained at the time when the two-dimensional image acquisition unit 62 has acquired the two-dimensional image (the visual sensor 30 captures the two-dimensional image). Using the height direction (Z-axis) information in the three-dimensional data, the distance [mm] from the visual sensor 30 to the upper surface of the corrugated fiberboard box (workpiece 81) can be determined, and the position information [mm] of the X and Y axes of the acquired corrugated cardboard box can be converted into position information [pixels] on the two-dimensional image.In particular, four corners (x1, y1), (x2, y1), (x1, y2), (x2, y2) of the corrugated cardboard box (workpiece 81) can be transformed into four corners (h1, w1), (h2, w1), (h1, w2), (h2, w2) on the two-dimensional image acquired by the two-dimensional image acquisition unit 62, and four corners on the image on the two-dimensional image can be defined as a processing area restricted by the two-dimensional image processing area restriction unit 64 to capture the MK label.It should be noted that the control device (control method) according to the present embodiment is not limited to application to a robot system that performs work by grasping a moving workpiece, or to a robot system that performs work by grasping a label that is attached to the upper surface of the moving workpiece, and can be comprehensively applied to a robot system that performs various processing operations on a moving workpiece.

[0047] Fig. Figure 10 is a flowchart to explain another control processing example by the robot controller (control device) in which in Fig. The function block diagram shown in section 3 should be noted. It should be noted that the diagram in Fig. The tax processing shown in section 10 corresponds to that which is based on reference to Fig. 5 is described, whereby the processes of steps ST3 and ST4 are replaced by ST31 to ST34. As described in Fig. Figure 10 shows that when a control processing example is started by the robot controller according to the first embodiment (START), in step ST1 the conveyor control device 5 controls a conveyor 6 to move the workpiece 81 in the direction indicated by the arrow 86. Fig. 6 is specified, and proceeds to step ST2. In step ST2, the generation unit 61 for three-dimensional information generates three-dimensional information based on the output of the visual sensor 30, the position detector 25 detects a first rotational position of a conveyor drive motor 24 and outputs the detected position to the conveyor control device 5 and then proceeds to step ST31.

[0048] In step ST31, an area of ​​a defined height (Z: vertical direction to an XY plane of the two-dimensional image) is read from a memory (memory unit 42), and furthermore the process continues to step ST32 to create an outer shape (current position) of the workpiece 81 from three-dimensional points (in the three-dimensional point group data) in an area (imaging area 35 in Fig. 8) to capture. It should be noted that the outer shape of the workpiece 81 in the XY plane can be defined as a circumscribed triangle that, for example, encloses a point (x, y) where the three-dimensional point is projected onto the XY plane. The three-dimensional point to be read from memory and processed can, for example, be data in a predetermined area in the vertical direction (Z-direction) with respect to the surface of the belt 6 of the conveyor 6. The area in the vertical direction can be conveniently defined according to, for example, the type, shape, and the like of the workpiece 81 handled by the robot system 100.Specifically, if the shape of the workpiece 81, which is handled by the robot system 100, is known in advance, it is possible, for example, by setting up the workpiece 81 max, which has the highest height in the workpiece 81, and the workpiece 81 min, which has the highest height in the imaging area 35, and capturing the workpiece 81 min, to set the height range without directly specifying the height range with the numerical value.

[0049] Next, in step ST33, a second rotational position of the conveyor drive motor 24 is calculated when the outer center point of the workpiece 81 is in the center of the field of view of the two-dimensional image with respect to the direction of movement of the conveyor 6, and the process continues to step ST34. In step ST34, the two-dimensional image is acquired at the time when the rotational position of the conveyor drive motor 24 is in its second rotational position, and the process continues to step ST5 to correct the three-dimensional information. The processes in steps ST6 to ST10 thereafter are the same as those described with reference to Fig. 5 are described, and their description is omitted. As described above, the processing area restriction unit 64 for two-dimensional image can restrict the processing area in the two-dimensional image acquired by the acquisition unit 62 for two-dimensional image, for example, based on the current position of the workpiece 81 in the three-dimensional information generated by the generation unit 61 for three-dimensional information. It should be noted that the control processing by the robot controller is not limited to the example described above and that various modifications and variations are possible.

[0050] Fig. Figure 11 is a top view schematically depicting an overall configuration of a second embodiment of a robot system according to the present embodiment. As can be seen from the comparison between Fig. 11 and Fig. As becomes clear in Figure 2, in a robot system 200 according to the second embodiment, a transport vehicle 7 is used instead of the conveyor 6 in the robot system 100 of the first embodiment. Specifically, in the robot system 200 according to the second embodiment, the transport vehicle 7 is an example of a mobile device that transports the workpiece 81. The workpiece 81 is placed on a placement table 7a and is automatically moved along a belt 39, which is, for example, attached to the floor surface. Specifically, the transport vehicle 7 includes a sensor that detects the belt 39 and moves automatically along the belt 39 while the belt 39 is being detected by the sensor.

[0051] As in Fig. Figure 11 shows a visual sensor 30, attached to a support element 37, positioned above a path along which the transport vehicle 7 moves. This visual sensor 30 is similar to the one in the robot system 100 according to the first embodiment described above and is positioned so that the workpiece 81, which is moved by the transport vehicle 7, can be imaged. It should be noted that the transport vehicle 7 includes a drive motor for drive wheels, and a position sensing means, corresponding to a position detector 25, can be attached to a conveyor drive motor 24 of the conveyor 6 on an output shaft of the drive motor. It should be noted that the placement table 7a is defined, for example, by the position of the setting point, which is fixed at any position of the transport vehicle 7, and that, for example, the control device 4, which is located in Fig.Figure 3 shows the position of the placement table 7a (workpiece 81).

[0052] Other configurations, measures, effects, and the like in the robot system 200 of the second embodiment are the same as those of the robot system 100 of the first embodiment described above, and descriptions thereof are omitted. As described above, the conveyor 6 and the transport vehicle 7 can be used as the mobile device for transporting the workpiece 81, but it is also possible, for example, to use a robot or the like for transporting the workpiece 81, without being limited to this. Furthermore, the imaging device 3 (visual sensor 30) moves the three-dimensional information in a predetermined coordinate system to acquire and synchronize the two-dimensional image and the three-dimensional information (three-dimensional point group data).As described above, according to the present embodiment it is possible to provide a control device, a robot system and a control method that are capable of performing high-speed processing with high accuracy.

[0053] Although the present disclosure is described in detail, it is not limited to the individual embodiments described above. These embodiments may be added, replaced, modified, partially omitted, or the like, within a scope that does not deviate from the essence of the present disclosure, or derived from the content described in the claims and their equivalents within a scope that does not deviate from the spirit of the present disclosure. These embodiments may also be implemented in combination. In the embodiment described above, for example, the sequence of each operation and the sequence of each processing step are given as an example and are not limited to this. This also applies if a numerical value or a mathematical expression is used in the description of the embodiment described above.

[0054] With regard to the embodiments and variations described above, the following descriptions are further disclosed. [Annex 1]

[0055] Control device (4) configured to receive an output from a visual sensor (30) that captures an image of a moving workpiece (81) and to control the processing of the workpiece (81), comprising the following: a two-dimensional image acquisition unit (62) configured to acquire a two-dimensional image that includes the workpiece (81) based on the output of the visual sensor (30); a three-dimensional information generation unit (61) configured to generate three-dimensional information including the workpiece (81) based on the output of the visual sensor (30); a processing area restriction unit (64) for a two-dimensional image, configured to restrict a processing area in a captured two-dimensional image; and a motion control unit (63) configured to change a relative position of the three-dimensional information based on restricting the processing area of ​​the two-dimensional image in a predetermined coordinate system. [Annex 2]

[0056] Control device according to Annex 1, wherein The processing area restriction unit (64) for two-dimensional image restricts the processing area in the acquired two-dimensional image based on the current position of the workpiece (81) in the generated three-dimensional information. [Annex 3]

[0057] Control device according to Annex 1 or 2, wherein The processing area restriction unit (64) for two-dimensional image restricts the processing area such that a border is added to an image area of ​​the workpiece (81) in the acquired two-dimensional image. [Annex 4]

[0058] Control device according to Annex 3, wherein The processing area restriction unit (64) for two-dimensional image restricts the processing area such that in the acquired two-dimensional image a front edge in a direction in which the workpiece (81) moves is wider than a rear edge in the direction in which the workpiece (81) moves. [Annex 5]

[0059] Control device according to Annex 4, wherein The processing area restriction unit (64) for two-dimensional image restricts the processing area such that in the acquired two-dimensional image, an edge on both sides of a direction in which the workpiece (81) moves is narrower than a rear edge in the direction in which the workpiece (81) moves. [Annex 6]

[0060] Control device according to Annex 1 or 2, wherein The processing area restriction unit (64) for two-dimensional image restricts the processing area such that the acquired two-dimensional image corresponds to an image area of ​​the workpiece (81). [Annex 7]

[0061] Control device according to one of Annexes 1 to 6, which further comprises the following: an operating control unit configured to control processing of the workpiece (81) based on a restricted processing area of ​​the two-dimensional image and the three-dimensional information. [Annex 8]

[0062] Control device according to one of Annexes 1 to 7, wherein The motion control unit calculates a movement quantity of the workpiece (81) based on a first position of the workpiece (81) when the output of the visual sensor (30) is used to generate the three-dimensional information, and a second position of the workpiece (81) when the output of the visual sensor (30) is used to acquire the two-dimensional image, and moves the three-dimensional information such that it corresponds to the movement quantity of the workpiece (81) in the coordinate system, thereby moving the three-dimensional information within a restricted processing area of ​​the two-dimensional image. [Annex 9]

[0063] Robot system (100, 200) comprising the following: a moving device configured to move the workpiece (81); the visual sensor (30) which is configured to image the moving workpiece (81); the driving control device according to one of Annexes 1 to 8; and a robot that is controlled by the control device. [Annex 10]

[0064] Robot system according to Annex 9, wherein the moving device is a conveyor (6) or a transport vehicle (7) configured to place and move the workpiece (81). [Annex 11]

[0065] Robot system according to Annex 9 or 10, wherein the visual sensor (30) is a sensor that is capable of capturing a two-dimensional image and three-dimensional information. [Annex 12]

[0066] Control method for controlling the processing on a workpiece (81) by receiving an output from a visual sensor (30) that captures an image of a moving workpiece (81) comprising the following: a step of raising a two-dimensional image that includes the workpiece (81) based on the output of the visual sensor (30); a step of generating three-dimensional information that includes the workpiece (81) based on the output of the visual sensor (30); a step of restricting a processing area in a raised two-dimensional image; and a step of changing a relative position of the three-dimensional information based on the fact that the processing area of ​​the two-dimensional image is restricted in a predetermined coordinate system. REFERENCE MARK LIST 1 robot 2 Hand 3 Imaging device 4 robot controllers (control device) 5 Conveyor control device 6 Conveyors (moving devices) 7 transport vehicles (mobile devices) 23, 25 Position detector 24 Conveyor drive motor 26 Conveyor drive device 30 visual sensors 31 First Camera 32 Second Camera 33 projector 41 Operational program 42, 52 storage 43, 53 Operational control unit 44 Hand drive unit 45 Robot drive 46 Display 47 Image processing unit 54 Conveyor drive unit 61 Image generation unit for three-dimensional information 62 units of measurement for a two-dimensional image 63 Motion control unit 64 Processing area restriction unit for two-dimensional image 65 Feature section recording unit 66 units of calculation 71 Measuring range 81 workpiece 91 Processing area 91a, 91b, 91c Marginal 100, 200 robot systems

Claims

A control device configured to receive output from a visual sensor capturing an image of a moving workpiece and to control the processing of the workpiece, comprising: a two-dimensional image acquisition unit configured to acquire a two-dimensional image that includes the workpiece based on the output of the visual sensor; a three-dimensional information generation unit configured to generate three-dimensional information that includes the workpiece based on the output of the visual sensor; a two-dimensional image processing area restriction unit configured to restrict a processing area in an acquired two-dimensional image;and a motion control unit configured to change the relative position of three-dimensional information based on restricting the processing area of ​​the two-dimensional image in a predetermined coordinate system. Control device according to claim 1, wherein the processing area restriction unit for a two-dimensional image restricts the processing area in the acquired two-dimensional image based on the current position of the workpiece in the generated three-dimensional information. Control device according to claim 1 or 2, wherein the processing area restriction unit for a two-dimensional image restricts the processing area in such a way that in the acquired two-dimensional image a border is added to an image area of ​​the workpiece in the acquired two-dimensional image. Control device according to claim 3, wherein the processing area restriction unit for a two-dimensional image restricts the processing area such that in the raised two-dimensional image a front edge in a direction in which the workpiece moves is wider than a rear edge in the direction in which the workpiece moves. Control device according to claim 4, wherein the processing area restriction unit for a two-dimensional image restricts the processing area such that in the raised two-dimensional image, an edge on both sides of a direction in which the workpiece moves is narrower than a rear edge in the direction in which the workpiece moves. Control device according to claim 1 or 2, wherein the processing area restriction unit for a two-dimensional image restricts the processing area such that an image area of ​​the workpiece corresponds in the acquired two-dimensional image. Control device according to one of claims 1 to 6, further comprising: an operating control unit configured to control processing of the workpiece based on a restricted processing area of ​​the two-dimensional image and the three-dimensional information. Control device according to one of claims 1 to 7, wherein: the motion control unit calculates a movement quantity of the workpiece based on a first position of the workpiece when the output of the visual sensor is used to generate the three-dimensional information, and a second position of the workpiece when the output of the visual sensor is used to acquire the two-dimensional image, and moves the three-dimensional information such that it corresponds to the movement quantity of the workpiece in the coordinate system, whereby the three-dimensional information moves within an area of ​​a restricted processing area of ​​the two-dimensional image. A robot system comprising: a moving device configured to move the workpiece; a visual sensor configured to image the moving workpiece; a control device according to any one of claims 1 to 8; and a robot controlled by the control device. Robot system according to claim 9, wherein the moving device is a conveyor or a transport vehicle configured to place and move the workpiece. Robot system according to claim 9 or 10, wherein the visual sensor is a sensor capable of acquiring a two-dimensional image and three-dimensional information. A control method for controlling processing on a workpiece by receiving an output from a visual sensor that captures an image of a moving workpiece, comprising: a step of acquiring a two-dimensional image that includes the workpiece, based on the output of the visual sensor; a step of generating three-dimensional information that includes the workpiece, based on the output of the visual sensor; a step of restricting a processing area in an acquired two-dimensional image; and a step of changing a relative position of the three-dimensional information based on the processing area of ​​the two-dimensional image being restricted in a predetermined coordinate system.

Images