Control device, robot system, and control method
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Filing Date
- 2025-11-10
- Publication Date
- 2026-05-22
Abstract
Description
Control device, robot system, and control method
[0001] The present disclosure relates to a control device, a robot system, and a control method.
[0002] Conventionally, there has been provided a control device (robot control device) that detects the surface shape and position of a workpiece (object) based on an image captured by a visual sensor (imaging device), and causes a robot to perform a predetermined task on the workpiece. Here, the visual sensor used is a two-dimensional sensor that captures a two-dimensional image of the workpiece, or a three-dimensional sensor that measures three-dimensional information such as the distance from the visual sensor to the workpiece.
[0003] Furthermore, conventionally, robot systems have been put to practical use in which a robot equipped with a work tool (end effector) is controlled by the above-mentioned control device to perform a predetermined task on a moving workpiece. In such robot systems, the control device calculates the position and orientation of the workpiece based on two-dimensional images and three-dimensional information acquired from the output of a visual sensor, and controls the robot's hand (work tool) to control a task such as gripping the workpiece. The workpiece is moved (transported) by, for example, a conveyor or a transport vehicle.
[0004] Incidentally, various control techniques have been proposed in the past for controlling a predetermined operation on a workpiece based on two-dimensional images and three-dimensional information acquired from the output of a visual sensor.
[0005] International Publication No. 2021-256437 Publication No. 05-173644
[0006] As mentioned above, conventionally, robot systems have been put into practical use that use, for example, two-dimensional images obtained from the output of a two-dimensional sensor and three-dimensional information generated from the output of a three-dimensional sensor to control a robot to perform a specified task on a moving workpiece.
[0007] However, in such conventional robot systems, for example, if the surface of the conveyor that transports the workpiece is mesh-like, there is a risk that the accuracy of detecting the workpiece and its characteristic features in the two-dimensional image will decrease. Furthermore, when performing detection processing based on two-dimensional images containing the workpiece and three-dimensional information, there are problems such as the amount of data to be processed increasing as the resolution of the visual sensor increases, requiring a high-performance processing unit, or making it difficult to perform detection processing at high speed.
[0008] Therefore, there is a demand for a control device, a robot system, and a control method that are capable of high-precision, high-speed processing.
[0009] According to one embodiment of the present disclosure, a control device is provided that receives the output of a visual sensor that images a moving workpiece and controls processing of the workpiece, the control device comprising a two-dimensional image acquisition unit, a three-dimensional information generation unit, a two-dimensional image processing range limitation unit, and a movement control unit.
[0010] The two-dimensional image acquisition unit acquires a two-dimensional image including the workpiece based on the output of the visual sensor, and the three-dimensional information generation unit generates three-dimensional information including the workpiece based on the output of the visual sensor. The two-dimensional image processing range limiting unit limits the processing range in the acquired two-dimensional image, and the movement control unit changes the relative position of the three-dimensional information based on the limited processing range of the two-dimensional image in a predetermined coordinate system.
[0011] FIG. 1 is a front view schematically illustrating the overall configuration of a first example of a robot system according to this embodiment. FIG. 2 is a plan view of the robot system shown in FIG. 1. FIG. 3 is a functional block diagram of the robot system shown in FIG. 1. FIG. 4 is a diagram illustrating an example of a visual sensor in the functional block diagram shown in FIG. 3. FIG. 5 is a flowchart illustrating an example of control processing by a robot control device (control device) in the functional block diagram shown in FIG. 3. FIG. 6 is a diagram illustrating how three-dimensional information of a workpiece is generated. FIG. 7 is a diagram illustrating an example of processing by a two-dimensional image processing range limiting unit in the functional block diagram shown in FIG. 3. FIG. 8 is a diagram illustrating an example of processing by a two-dimensional image acquisition unit in the functional block diagram shown in FIG. 3. FIG. 9 is a diagram illustrating another example of processing by the two-dimensional image processing range limiting unit in the functional block diagram shown in FIG. 3. FIG. 10 is a flowchart illustrating another example of control processing by a robot control device (control device) in the functional block diagram shown in FIG. 3. FIG. 11 is a plan view schematically illustrating the overall configuration of a second example of a robot system according to this embodiment.
[0012] Hereinafter, examples of a control device, a robot system, and a control method according to the present embodiment will be described in detail with reference to the accompanying drawings. In each drawing, the same or similar components are assigned the same or similar reference numerals. Furthermore, the embodiments described below do not limit the technical scope and meaning of the terms of the invention described in the claims.
[0013] Fig. 1 is a front view showing a schematic overall configuration of a first example of a robot system according to this embodiment, and Fig. 2 is a plan view of the robot system shown in Fig. 1. In Fig. 1 and Fig. 2, reference numeral 100 indicates the robot system of the first example, with 1 indicating a robot, 2 indicating a hand, 3 indicating an imaging device, 6 indicating a conveyor, and 81 indicating a workpiece (object). As shown in Fig. 1 and Fig. 2, the robot system 100 includes a robot 1 equipped with a hand 2, a conveyor 6, and an imaging device 3.
[0014] The robot 1 includes a base 14 fixed to an installation surface, a swivel base 13 that rotates relative to the base 14, a lower arm 12 rotatably supported on the swivel base 13, an upper arm 11 rotatably supported on the lower arm 12, and a wrist 15 rotatably supported at the end of the upper arm 11. The upper arm 11 rotates around a rotation axis parallel to the direction of extension of the upper arm 11, and a rotatably formed flange 16 is provided at the tip of the wrist 15. In the description of this embodiment, the robot 1 is described as an articulated robot having multiple joints, but this is not limited to this configuration, and any robot capable of controlling various work tools can be applied. Furthermore, while the workpiece 81 is depicted as a rectangular parallelepiped cardboard box by way of example, the workpiece 81 is, of course, not limited to a cardboard box.
[0015] The hand 2 is a work tool capable of gripping a workpiece 81, and is configured to, for example, suck and grip the workpiece 81 using a plurality of suction pads 2a. The hand 2 is fixed to a flange 16 of the wrist 15, but any work tool can be applied as the work tool attached to the robot 1.
[0016] The conveyor 6 is an example of a moving device that transports the workpiece 81, and transports the workpiece 81 in a predetermined direction by rotating a circular belt 6a. That is, the conveyor 6 moves the workpiece 81 horizontally as shown by the arrow 86, and transports the workpiece 81 to a position where the robot 1 can change its position and posture and the hand 2 can grasp the workpiece 81.
[0017] The imaging device 3 includes a two-dimensional sensor (visual sensor 30) that captures a two-dimensional image of the workpiece 81, and a three-dimensional sensor (visual sensor 30) that acquires three-dimensional information about the surface of the workpiece 81. That is, based on the output of the visual sensor 30, it is possible to capture a two-dimensional image of the workpiece 81 being transported by the conveyor 6 and acquire three-dimensional information about the surface of the workpiece 81 (three-dimensional point cloud data, three-dimensional position information). Here, reference numeral 24 denotes a conveyor drive motor that drives the circular belt 6a of the conveyor 6, and 25 denotes a position detector such as an encoder. An example of the visual sensor 30 will be described in detail later with reference to FIG. 4.
[0018] The visual sensor 30 is supported by a support member 37 and is positioned so as to capture an image of the workpiece 81 being transported by the conveyor 6. The visual sensor 30 is positioned upstream of the robot 1 in the direction in which the workpiece 81 is transported. A world coordinate system 76 is set in the robot system 100 as a reference coordinate system, and the origin of the world coordinate system 76 is located, for example, on the base 14 of the robot 1. This ensures that the position and orientation of the world coordinate system 76 do not change even if the position and posture of the robot 1 change. The world coordinate system 76 has X-, Y-, and Z-axes that are orthogonal to each other as its coordinate axes, and W-, P-, and R-axes are set as coordinate axes around the X-, Y-, and Z-axes.
[0019] Furthermore, in the robot system 100, a tool coordinate system 77 is set whose origin is set at an arbitrary position on the work tool, and the origin of the tool coordinate system 77 is set to the tool center point of the hand 2. As a result, when the position and orientation of the robot 1 change, the position and orientation of the tool coordinate system 77 also change; for example, the position of the robot 1 corresponds to the position of the tool center point, and the orientation of the robot 1 corresponds to the orientation of the tool coordinate system 77 with respect to the world coordinate system 76. Furthermore, in the robot system 100, a sensor coordinate system 78 is set corresponding to the visual sensor 30, and the origin of the sensor coordinate system 78 is a coordinate system fixed to the visual sensor 30. As a result, coordinate values in the sensor coordinate system 78 can be converted to coordinate values in the world coordinate system 76 based on the position and orientation of the sensor coordinate system 78 with respect to the world coordinate system 76.
[0020] 3 is a functional block diagram of the robot system shown in FIG. 1, and the robot system 100 further includes a robot control device (control device) 4 and a conveyor control device 5. As shown in FIGS. 1 to 3, the robot 1 includes a plurality of robot drive motors 22 that change the position and posture of the robot 1, and a position detector 23 such as an encoder attached to each robot drive motor 22. The hand 2 includes, for example, a pump 21 that reduces the pressure inside the suction pad 2a to suck the workpiece 81.
[0021] The robot control device 4 includes a storage unit 42, a motion control unit 43, a hand driving unit 44, a robot driving unit 45, a display 46, and an image processing unit 47. The motion control unit 43 and the image processing unit 47 are configured by a microprocessor (MPU) that functions as an arithmetic processing device, and perform various processes by exchanging data with the storage unit 42, which may be a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory. An operation program 41 that controls the robot 1, the hand 2, the conveyor 6, etc. is stored in the storage unit 42 and is executed by the arithmetic processing device (the motion control unit 43, the image processing unit 47, etc.).
[0022] The operation control unit 43 outputs operation commands according to the operation program 41 to the hand driving unit 44 and the robot driving unit 45, thereby controlling the hand 2 and the robot 1. The operation control unit 43 also outputs operation commands according to the operation program 41 to the image processing unit 47, thereby controlling the acquisition of two-dimensional images based on the output of the visual sensor 30, the generation of three-dimensional information, image processing, etc. The display 46 is formed, for example, by a liquid crystal display panel, and displays information relating to the control of the hand 2, the robot 1, and the conveyor 6, etc.
[0023] The conveyor control device 5 includes a memory unit 52, an operation control unit 53, and a conveyor drive unit 54, and controls the conveyor 6 based on commands from the robot control device 4 to transport the workpiece 81 to a predetermined position. The conveyor 6 is configured to include a conveyor drive motor 24 and a position detector 25. The position detector 25 can be configured, for example, as an encoder (rotary encoder) attached to the output shaft of the conveyor drive motor 24, and the approximate position of the workpiece 81 being transported by the conveyor 6 can be recognized based on the output of this encoder. The position detector 25 is not limited to an encoder, and a photoelectric sensor, for example, can also be used.
[0024] The image processing unit 47 includes a three-dimensional information generation unit 61, a two-dimensional image acquisition unit 62, a movement control unit 63, a two-dimensional image processing range limiting unit 64, a characteristic part detection unit 65, and a calculation unit 66. The three-dimensional information generation unit 61 generates three-dimensional information based on the output of the three-dimensional sensor, and the two-dimensional image acquisition unit 62 acquires a two-dimensional image based on the output of the two-dimensional sensor. The movement control unit 63 corrects the relative position of the three-dimensional information with respect to the two-dimensional image, and the two-dimensional image processing range limiting unit 64 limits the two-dimensional image to a narrower range from the two-dimensional image acquired by the two-dimensional image acquisition unit 62 and outputs the limited two-dimensional image as the processing range.
[0025] The characteristic part detection unit 65 detects predetermined characteristic parts of the workpiece 81, and the calculation unit 66 calculates the position and posture of the robot 1 based on the position and posture of the workpiece 81. Here, the image processing unit 47 does not have to be built into the robot control device 4, but may be provided as a separate unit outside the robot control device 4, and the conveyor control device 5 may also be built into the robot control device 4, allowing for various changes and modifications.
[0026] FIG. 4 is a diagram illustrating an example of a visual sensor in the functional block diagram shown in FIG. 3 . As shown in FIG. 4 , the visual sensor 30 (imaging device 3) includes a first camera 31, a second camera 32, and a projector 33. Here, the first camera 31 and the second camera 32 function as stereo cameras, but each camera 31 or 32 also functions as a two-dimensional camera that captures two-dimensional images. Note that any cameras equipped with an imaging element such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor can be used as the cameras 31 and 32. Furthermore, the two cameras 31 and 32 are disposed at predetermined positions spaced apart from each other.
[0027] The projector 33 projects, for example, a pattern light such as a striped pattern onto the workpiece 81, and the cameras 31, 32 and the projector 33 are disposed inside the housing 34. Here, when the three-dimensional information generation unit 61 generates three-dimensional information based on the output of the visual sensor 30, for example, the projector 33 projects the pattern light and uses stereo images (two two-dimensional images) captured by the first camera 31 and the second camera 32. Furthermore, when the two-dimensional image acquisition unit 62 acquires a two-dimensional image based on the output of the visual sensor 30, for example, the pattern light from the projector 33 is stopped and a two-dimensional image captured by either the first camera 31 or the second camera 32 is used. Therefore, it is preferable that the timing at which the three-dimensional information generation unit 61 processes the output of the visual sensor 30 be different from the timing at which the two-dimensional image acquisition unit 62 acquires the output of the visual sensor 30. However, if the visual sensor 30 is equipped with both a dedicated three-dimensional sensor for outputting to the three-dimensional information generating unit 61, such as a TOF (Time Of Flight) type, and a dedicated two-dimensional sensor for outputting to the two-dimensional image acquiring unit 62, it is possible to perform processing without any difference in timing. Furthermore, it goes without saying that various known three-dimensional sensors and two-dimensional sensors other than those described above can be used as the visual sensor 30.
[0028] 3 and 4, the three-dimensional information generator 61 processes the image acquired by the visual sensor 30 to generate three-dimensional information of the surface of the imaging target as three-dimensional point cloud data (three-dimensional map). Here, the three-dimensional information includes, for example, information about the positions of multiple measurement points set on the surface of the imaging target. The three-dimensional point cloud data represents the position of the surface of the imaging target using a set of coordinate values (x, y, z) of the measurement points set on the surface of the imaging target.
[0029] For example, the three-dimensional information generator 61 sets multiple measurement points on the surface of the object to be imaged within the imaging range 35 of the visual sensor 30. These measurement points can be set for each pixel of the two-dimensional image captured by the camera 31 or the camera 32. The three-dimensional information generator 61 can also calculate the distance from the visual sensor 30 to each measurement point based on the parallax between the two-dimensional images captured by the two cameras 31 and 32. The three-dimensional information generator 61 then calculates the coordinate values of the measurement points in the sensor coordinate system 78 based on the distance from the visual sensor 30 to each measurement point. Alternatively, the three-dimensional information generator 61 can convert the coordinate values in the sensor coordinate system 78 into coordinate values in the world coordinate system 76 based on the position and orientation of the visual sensor 30. In this way, the three-dimensional information generator 61 generates three-dimensional point cloud data (three-dimensional information) including the coordinate values of the multiple measurement points.
[0030] The movement control unit 63 acquires the output of the position detector 25 of the conveyor 6 and calculates the actual movement amount of the workpiece 81 from the time the stereo image was captured to the time the two-dimensional image was captured in order to generate three-dimensional information. The movement control unit 63 then controls the movement of the three-dimensional information so that it corresponds to the two-dimensional image in a predetermined coordinate system, corresponding to the movement amount of the workpiece 81. This control makes it possible to superimpose at least a portion of the three-dimensional information of the workpiece 81 on the two-dimensional image of the workpiece 81 in the above-mentioned coordinate system. In other words, it is possible to generate three-dimensional information and two-dimensional images equivalent to three-dimensional information and two-dimensional images acquired at the same time.
[0031] Fig. 5 is a flowchart for explaining an example of control processing by the robot control device (control device) in the functional block diagram shown in Fig. 3. As shown in Fig. 5, when an example of control processing by the robot control device of the first embodiment starts (START), in step ST1, the conveyor control device 5 controls the conveyor 6 to move the workpiece 81 in the direction indicated by the arrow 86 in Fig. 6 so that the workpiece 81 is included in the imaging range 35 of the visual sensor 30. Whether the workpiece 81 is included in the imaging range 35 can be recognized based on the output of the position detector 25.
[0032] Next, the process proceeds to step ST2, where the three-dimensional information generator 61 generates three-dimensional information based on the output (e.g., stereo image) of the visual sensor 30. At this time, the position detector 25 detects the first rotation position of the conveyor drive motor 24 and outputs it to the conveyor control device 5. Further, the process proceeds to step ST3, where the two-dimensional image acquirer 62 acquires a two-dimensional image based on the output (e.g., two-dimensional image captured by the first camera 31) of the visual sensor 30. At this time, the position detector 25 detects the second rotation position of the conveyor drive motor 24 and outputs it to the conveyor control device 5.
[0033] Here, a sensor (e.g., a photoelectric sensor) may be provided on the conveyor 6 to detect that the workpiece 81 has arrived within the imaging range 35, and the rotational positions (first and second rotational positions) of the position detector 25 may be detected based on the output of the sensor. The first rotational position and second rotational position output from the position detector 25 are stored and held in the memory unit 52, for example.
[0034] The process then proceeds to step ST4, where the movement distance of the workpiece 81 on the conveyor 6 is calculated, and the process proceeds to step ST5. In step ST5, the movement control unit 63 corrects the relative position of the three-dimensional information with respect to the two-dimensional image, and the process proceeds to step ST6. In step ST6, the two-dimensional image processing range limiting unit 64 calculates (limits) a processing range from the two-dimensional image acquired by the two-dimensional image acquisition unit 62, and the process proceeds to step ST7. The processing range limiting process by the two-dimensional image processing range limiting unit 64 will be described in detail later with reference to FIGS. 6 to 9.
[0035] In step ST7, the characteristic portion detection unit 65 detects the characteristic portion of the workpiece 81 in the two-dimensional image whose processing range has been limited by the two-dimensional image processing range limiting unit 64. Here, for example, if the top surface of the workpiece 81 is set as the characteristic portion, a reference image of the top surface of the workpiece 81 is stored in advance in the storage unit 42. The reference image can be an image of the top surface of the workpiece 81 actually captured by a two-dimensional camera, or the reference image of the workpiece 81 can be generated based on three-dimensional data obtained by a CAD (Computer Aided Design) device. The characteristic portion detection unit 65 detects the image of the top surface of the workpiece 81 in the two-dimensional image whose processing range has been limited by the two-dimensional image processing range limiting unit 64 by a method such as template matching using the reference image.
[0036] Thus, according to the control device (robot system) of this embodiment, even if the surface of the conveyor 6 (belt 6a) is mesh-like or the like, processing is performed based on a two-dimensional image whose processing range is limited by the two-dimensional image processing range limiting unit 64, thereby reducing erroneous detection of the workpiece 81, for example. Furthermore, the processing range of the two-dimensional image limited by the two-dimensional image processing range limiting unit 64 has a narrower processing region (area) than the two-dimensional image acquired by the two-dimensional image acquiring unit 62, making it possible to speed up processing.
[0037] Next, the process proceeds to step ST8, where the calculation unit 66 calculates the position and orientation of the workpiece 81. Here, the position of the workpiece 81 can be set to, for example, the center of gravity of the rectangle on the top surface of the workpiece 81, a specific position, or the position of an end, and the orientation of the workpiece can be set to, for example, the normal direction of the top surface of the workpiece 81. The calculation unit 66 extracts measurement points located in an area overlapping the image 72a, and calculates the position and orientation of the workpiece 81 based on the coordinate values of the multiple measurement points.
[0038] The process then proceeds to step ST9, where the calculation unit 66 calculates the position and orientation of the robot 1 based on the position and orientation of the workpiece 81. That is, the calculation unit 66 calculates the position and orientation of the workpiece 81 when the workpiece 81 is moved by the conveyor 6 to a position where it will be grasped by the hand 2, and calculates the position and orientation of the robot 1 based on the position and orientation of the workpiece 81 at that time.
[0039] Then, the process proceeds to step ST10, where the calculation unit 66 sends a command to the operation control unit 43 to drive the robot 1 and the hand 2. The operation control unit 43 drives the robot 1 and the hand 2 based on the command from the calculation unit 66, and the robot 1 (hand 2) grasps and transports the workpiece 81, and an example of control processing by the robot control device of this first embodiment is ended (END).
[0040] Thus, according to one example of control processing by the robot control device of the first embodiment, the two-dimensional image processing range limiting unit 64 limits the processing range from the two-dimensional image acquired by the two-dimensional image acquiring unit 62, thereby performing detection processing and the like based on the actually required two-dimensional image, thereby reducing erroneous detection of the workpiece 81. Furthermore, since the processing range of the two-dimensional image limited by the two-dimensional image processing range limiting unit 64 has a smaller area than the two-dimensional image acquired by the two-dimensional image acquiring unit 62, processing can be sped up.
[0041] FIG. 6 is a diagram illustrating the generation of three-dimensional information (three-dimensional point cloud data) of a workpiece, and FIG. 7 is a diagram illustrating an example of processing by the two-dimensional image processing range limiting unit in the functional block diagram shown in FIG. 3. First, as shown in FIG. 6, a workpiece 81 is transported by a conveyor 6 in the direction of arrow 86, and an imaging range 35 including the workpiece 81 is imaged by the visual sensor 30. In FIG. 7, reference numeral 71 denotes a two-dimensional image (measurement area) acquired by the two-dimensional image acquisition unit 62, with 71a representing the image area of the workpiece 81, 71b representing image areas (measurement points PX) at both ends of the belt 6a of the conveyor 6, 71c representing the image area of the transport portion of the belt 6a, and 71d representing the image area of the floor on which the conveyor 6 is installed. Furthermore, reference numeral 91 denotes a processing range limited (calculated) by the two-dimensional image processing range limiting unit 64, and 91a to 91c represent margins.
[0042] 7, the two-dimensional image processing range limiting unit 64 adds margins 91a to 91c to the image area 71a of the workpiece 81 in the two-dimensional image 71 acquired by the two-dimensional image acquiring unit 62, and limits it as a processing range 91 for actual processing. Here, the processing range 91 is defined so that the front margin 91c in the direction in which the workpiece 81 moves is wider than the rear margin 91a in the direction in which the workpiece 81 moves, and further so that the margins 91b, 91b on both sides in the direction in which the workpiece 81 moves are narrower than the rear margin 91a in the direction in which the workpiece 81 moves.
[0043] That is, since margin 91c is the portion before the arrival of workpiece 81, it is considered preferable to set it wider than margin 91a, which is the portion after the arrival of workpiece 81. Also, since margin 91b hardly deviates in the direction perpendicular to the movement direction of workpiece 81, it is considered acceptable to set it narrower than margin 91a, which is the portion after the arrival of workpiece 81. Note that the processing range limiting process by two-dimensional image processing range limiting unit 64 is not limited to the one described above.
[0044] FIG. 8 is a diagram illustrating an example of processing by the 2D image acquisition unit in the functional block diagram shown in FIG. 3 . In FIG. 8 , the position of the workpiece 81 used to generate the 3D information is indicated by a dashed line. The workpiece 81 moves in the direction indicated by the arrow 86 by the conveyor 6. For example, the first camera 31 of the visual sensor 30 captures a 2D image of the workpiece 81 when the workpiece 81 is transported within the imaging range 35. That is, the visual sensor 30 can capture the 2D image immediately after capturing the image for generating the 3D information. Alternatively, the visual sensor 30 can capture the 2D image a predetermined time after capturing the image for acquiring the 3D information.
[0045] 8, the movement control unit 63 controls the movement of each measurement point, and the three-dimensional information includes, for example, the coordinate value of measurement point P1A located on the surface of the workpiece 81. The movement control unit 63 processes measurement point P1A in the direction in which the workpiece 81 is moving based on the amount of movement of the workpiece 81, and processes measurement points P1A and P2A to move to measurement points P1B and P2B. Here, the movement control unit 63 changes the relative position of the three-dimensional information generated by the three-dimensional information generation unit 61 based on, for example, the processing range of the two-dimensional image limited by the two-dimensional image processing range limiting unit 64 in a predetermined coordinate system (for example, the sensor coordinate system 78 shown in FIG. 1).
[0046] FIG. 9 is a diagram illustrating another example of processing by the two-dimensional image processing range limiting unit in the functional block diagram shown in FIG. 3. The processing range limiting process by the two-dimensional image processing range limiting unit 64 described with reference to FIG. 7 limits the processing range 91 by providing margins 91a-91c around the image area 71a of the workpiece 81. However, the processing range limiting process by the two-dimensional image processing range limiting unit 64 described with reference to FIG. 9 limits the processing range to coincide with the image area of the workpiece 81 in the two-dimensional image acquired by the two-dimensional image acquisition unit 62. In FIG. 9, reference symbol MK denotes a label, such as a two-dimensional code, attached to the top surface of the workpiece 81, e.g., a cardboard box, and reference symbol 79 denotes an image coordinate system. Reference symbols 72, 72a, 72b, and 72c denote parts corresponding to 71, 71a, 71b, and 71c in FIG. 7, and reference symbol 87 denotes the direction of movement of the workpiece 81.
[0047] 9 , for example, when a label MK affixed to the top surface of a workpiece 81 is read by the visual sensor 30 to perform a predetermined task, the two-dimensional image processing range limiting unit 64 limits the image area 72a (71a) of the workpiece 81 in the two-dimensional image 72 acquired by the two-dimensional image acquiring unit 62 as the processing range for actual processing. That is, for example, when a label MK affixed to the top surface of the workpiece 81 is read, it is considered possible to read the label MK even if the processing range limited by the two-dimensional image processing range limiting unit 64 is the image area of the workpiece 81.
[0048] Specifically, by correcting the three-dimensional information in step ST5 of the flowchart shown in Fig. 5, for example, three-dimensional information (three-dimensional position information) at the timing when the two-dimensional image acquisition unit 62 acquires the two-dimensional image (when the visual sensor 30 captures the two-dimensional image) can be obtained. The height direction (Z axis) information in this three-dimensional information can be used to obtain the distance [mm] from the visual sensor 30 to the top surface of the cardboard box (workpiece 81), and the obtained X and Y position information [mm] of the cardboard box can be converted into position information [pixel] on the two-dimensional image. That is, the four corners (x1, y1), (x2, y1), (x1, y2), and (x2, y2) of the cardboard box (workpiece 81) are converted into four corners (h1, w1), (h2, w1), (h1, w2), and (h2, w2) on the two-dimensional image acquired by the two-dimensional image acquisition unit 62, and the four corners on this two-dimensional image can be set as the processing range to be limited by the two-dimensional image processing range limiting unit 64 in order to detect the label MK. Note that the control device (control method) according to this embodiment is not limited to application to a robot system that grips a moving workpiece to perform work, or a robot system that detects a label affixed to the top surface of a moving workpiece to perform work, but can be widely applied to robot systems that perform various processes on moving workpieces.
[0049] FIG. 10 is a flowchart illustrating another example of control processing by the robot control device (control device) in the functional block diagram shown in FIG. 3. The control processing illustrated in FIG. 10 corresponds to the control processing described with reference to FIG. 5, where steps ST3 and ST4 are replaced with steps ST31 to ST34. As shown in FIG. 10, when this example of control processing by the robot control device of the first embodiment starts (START), in step ST1, the conveyor control device 5 controls the conveyor 6 to move the workpiece 81 in the direction indicated by arrow 86 in FIG. 6, and the process proceeds to step ST2. In step ST2, the three-dimensional information generator 61 generates three-dimensional information based on the output of the visual sensor 30, and the position detector 25 detects the first rotational position of the conveyor drive motor 24 and outputs the information to the conveyor control device 5. The process then proceeds to step ST31.
[0050] In step ST31, the specified height (Z: perpendicular to the XY plane of the two-dimensional image) range is read from memory (storage unit 42). Then, the process proceeds to step ST32, where the outline (current position) of the workpiece 81 is detected from three-dimensional points (three-dimensional point cloud data) within the range (imaging range 35 in FIG. 8 ). Here, the outline of the workpiece 81 in the XY plane can be defined, for example, as a circumscribed rectangle including points (x, y) obtained by projecting the three-dimensional points onto the XY plane. Furthermore, the three-dimensional points read from memory and processed can be, for example, data of a predetermined range in the height (Z) direction relative to the surface of the belt 6a of the conveyor 6. This height range can be set in advance as appropriate, for example, depending on the type and shape of the workpiece 81 handled by the robot system 100. Specifically, if the shape of the workpiece 81 handled by the robot system 100 is known in advance, for example, the tallest workpiece 81max and the shortest workpiece 81min among the workpieces 81 handled can be placed within the imaging range 35 and detected, making it possible to set the height range without directly specifying a numerical value.
[0051] Next, the process proceeds to step ST33, where the second rotation position of the conveyor drive motor 24 is calculated when the center of the workpiece 81's outline relative to the direction of movement of the conveyor 6 becomes the center of the field of view of the two-dimensional image, and then the process proceeds to step ST34. In step ST34, a two-dimensional image is acquired when the rotation position of the conveyor drive motor 24 reaches the second rotation position, and the process proceeds to step ST5, where the three-dimensional information is corrected. Note that the subsequent processes of steps ST6 to ST10 are similar to those described with reference to FIG. 5 , and therefore their description will be omitted. In this way, the two-dimensional image processing range limiting unit 64 can limit the processing range of the two-dimensional image acquired by the two-dimensional image acquiring unit 62, for example, based on the current position of the workpiece 81 in the three-dimensional information generated by the three-dimensional information generating unit 61. It goes without saying that the control process by the robot control device is not limited to the above example and various modifications and variations are possible.
[0052] FIG. 11 is a plan view schematically illustrating the overall configuration of a second embodiment of a robot system according to this embodiment. As is clear from a comparison of FIG. 11 with FIG. 2 described above, a robot system 200 according to this second embodiment uses a transport vehicle 7 instead of the conveyor 6 in the robot system 100 according to the first embodiment. That is, in the robot system 200 according to the second embodiment, the transport vehicle 7 is an example of a moving device that transports a workpiece 81. The workpiece 81 is placed on a mounting table 7a, and the transport vehicle 7 automatically moves along a tape 39 affixed to the floor, for example. That is, the transport vehicle 7 includes a sensor that detects the tape 39, and automatically moves along the tape 39 while detecting the tape 39 using the sensor.
[0053] As shown in Figure 11, a visual sensor 30 fixed to a support member 37 is disposed above the path along which the transport vehicle 7 moves. This visual sensor 30 is similar to that in the robot system 100 of the first embodiment described above, and is disposed in a position where it can capture an image of the workpiece 81 being moved by the transport vehicle 7. Here, the transport vehicle 7 includes a drive motor for driving the wheels, and a position detector equivalent to the position detector 25 attached to the conveyor drive motor 24 of the conveyor 6 is attached to the output shaft of the drive motor. The position of the mounting table 7a can be determined, for example, by the position of a set point set at an arbitrary position on the transport vehicle 7, and for example, the control device 4 shown in Figure 3 can recognize the position of the mounting table 7a (workpiece 81).
[0054] The other configurations, operations, and effects of the robot system 200 of the second embodiment are similar to those of the robot system 100 of the first embodiment, and therefore a description thereof will be omitted. As described above, the conveyor 6 and the transport vehicle 7 can be used as the moving device for transporting the workpiece 81. However, the present invention is not limited to these. For example, a robot for transporting the workpiece 81 can also be used. Furthermore, various imaging devices (visual sensors 30) can be used as long as they can acquire and synchronize two-dimensional images and three-dimensional information (three-dimensional point cloud data) by moving three-dimensional information in a predetermined coordinate system. Thus, according to this embodiment, a control device, a robot system, and a control method capable of high-precision, high-speed processing can be provided.
[0055] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values or mathematical expressions are used in the description of the above-described embodiments.
[0056] The following supplementary notes are further disclosed regarding the above-described embodiments and modified examples. [Supplementary Note 1] A control device (4) that receives output from a visual sensor (30) that captures an image of a moving workpiece (81) and controls processing of the workpiece (81), comprising: a two-dimensional image acquisition unit (62) that acquires a two-dimensional image including the workpiece (81) based on the output of the visual sensor (30); a three-dimensional information generation unit (61) that generates three-dimensional information including the workpiece (81) based on the output of the visual sensor (30); a two-dimensional image processing range limiting unit (64) that limits a processing range in the acquired two-dimensional image; and a movement control unit (63) that changes the relative position of the three-dimensional information based on the limited processing range of the two-dimensional image in a predetermined coordinate system. [Supplementary Note 2] The control device according to Supplementary Note 1, wherein the two-dimensional image processing range limiting unit (64) limits the processing range in the acquired two-dimensional image based on the current position of the workpiece (81) in the generated three-dimensional information. [Supplementary Note 3] The control device according to Supplementary Note 1 or Supplementary Note 2, wherein the two-dimensional image processing range limiting unit (64) limits the processing range by adding a margin to an image area of the workpiece (81) in the acquired two-dimensional image. [Supplementary Note 4] The control device according to Supplementary Note 3, wherein the two-dimensional image processing range limiting unit (64) limits the processing range in the acquired two-dimensional image so that a front margin in a direction in which the workpiece (81) moves is wider than a rear margin in the direction in which the workpiece (81) moves. [Supplementary Note 5] The control device according to Supplementary Note 4, wherein the two-dimensional image processing range limiting unit (64) limits the processing range in the acquired two-dimensional image so that margins on both sides in a direction in which the workpiece (81) moves are narrower than a rear margin in the direction in which the workpiece (81) moves. [Supplementary Note 6] The control device according to Supplementary Note 1 or Supplementary Note 2, wherein the two-dimensional image processing range limiting unit (64) limits a processing range in the acquired two-dimensional image so as to coincide with an image area of the workpiece (81).[Supplementary Note 7] The control device according to any one of Supplementary Notes 1 to 6, further comprising an operation control unit that controls processing of the workpiece (81) based on the limited processing range of the two-dimensional image and the three-dimensional information. [Supplementary Note 8] The control device according to any one of Supplementary Notes 1 to 7, wherein the movement control unit calculates a movement amount of the workpiece (81) based on a first position of the workpiece (81) when using output of the visual sensor (30) to generate the three-dimensional information and a second position of the workpiece (81) when using output of the visual sensor (30) to acquire the two-dimensional image, and moves the three-dimensional information so as to correspond to the movement amount of the workpiece (81) in the coordinate system, thereby moving the three-dimensional information within an area of the limited processing range of the two-dimensional image. [Supplementary Note 9] A robot system (100, 200) comprising: a moving device that moves the workpiece (81); the visual sensor (30) that captures an image of the moving workpiece (81); the control device described in any one of Supplementary Note 1 to Supplementary Note 8; and a robot controlled by the control device. [Supplementary Note 10] The robot system described in Supplementary Note 9, wherein the moving device is a conveyor (6) or a transport vehicle (7) that moves while carrying the workpiece (81). [Supplementary Note 11] The robot system described in Supplementary Note 9 or Supplementary Note 10, wherein the visual sensor (30) is a sensor that can acquire two-dimensional images and three-dimensional information. [Supplementary Note 12] A control method for receiving an output from a visual sensor (30) that captures an image of a moving workpiece (81) and controlling processing of the workpiece (81), the control method comprising the steps of: acquiring a two-dimensional image including the workpiece (81) based on the output of the visual sensor (30); generating three-dimensional information including the workpiece (81) based on the output of the visual sensor (30); limiting a processing range in the acquired two-dimensional image; and changing a relative position of the three-dimensional information based on the limited processing range of the two-dimensional image in a predetermined coordinate system.
[0057] REFERENCE SIGNS LIST 1 Robot 2 Hand 3 Imaging device 4 Robot control device (control device) 5 Conveyor control device 6 Conveyor (moving device) 7 Transport vehicle (moving device) 23, 25 Position detector 24 Conveyor drive motor 26 Conveyor drive device 30 Visual sensor 31 First camera 32 Second camera 33 Projector 41 Operation program 42, 52 Memory unit 43, 53 Operation control unit 44 Hand drive unit 45 Robot drive unit 46 Display 47 Image processing unit 54 Conveyor drive unit 61 Three-dimensional information generation unit 62 Two-dimensional image acquisition unit 63 Movement control unit 64 Two-dimensional image processing range limiting unit 65 Characteristic portion detection unit 66 Calculation unit 71 Measurement area 81 Workpiece 91 Processing range 91a, 91b, 91c Margin 100, 200 Robot system
Claims
1. A control device that receives the output of a vision sensor that images a moving workpiece and controls processing on the workpiece, A two-dimensional image acquisition unit acquires a two-dimensional image including the workpiece based on the output of the aforementioned vision sensor, A three-dimensional information generation unit generates three-dimensional information including the workpiece based on the output of the visual sensor, A two-dimensional image processing range limiting unit that limits the processing range in the acquired two-dimensional image, The system includes a movement control unit that changes the relative position of the three-dimensional information based on a limited processing range of the two-dimensional image in a predetermined coordinate system, Control device.
2. The two-dimensional image processing range limiting unit limits the processing range in the acquired two-dimensional image based on the current position of the workpiece in the generated three-dimensional information. The control device according to claim 1.
3. The aforementioned two-dimensional image processing range limiting unit limits the processing range by adding a margin to the image area of the workpiece in the acquired two-dimensional image. The control device according to claim 1 or claim 2.
4. The two-dimensional image processing range limiting unit limits the processing range in the acquired two-dimensional image such that the margin in front of the direction in which the workpiece moves is wider than the margin behind the direction in which the workpiece moves. The control device according to claim 3.
5. The two-dimensional image processing range limiting unit limits the processing range in the acquired two-dimensional image such that the margins on both sides in the direction in which the workpiece moves are narrower than the margin behind the direction in which the workpiece moves. The control device according to claim 4.
6. The aforementioned two-dimensional image processing range limiting unit limits the processing range in the acquired two-dimensional image to match the image area of the workpiece. The control device according to claim 1 or claim 2.
7. Furthermore, it includes an operation control unit that controls processing on the workpiece based on the limited two-dimensional image processing range and the three-dimensional information. The control device according to claim 1 or claim 2.
8. The movement control unit calculates the amount of movement of the workpiece based on the first position of the workpiece when using the output of the vision sensor to generate the three-dimensional information and the second position of the workpiece when using the output of the vision sensor to acquire the two-dimensional image, and moves the three-dimensional information in the coordinate system to correspond to the amount of movement of the workpiece, thereby moving the three-dimensional information within the limited processing range of the two-dimensional image. The control device according to claim 1 or claim 2.
9. A moving device for moving the aforementioned workpiece, The visual sensor captures the moving workpiece, A control device according to claim 1 or claim 2, A robot controlled by the aforementioned control device, Robot system.
10. The aforementioned moving device is a conveyor or transport vehicle on which the workpiece is placed and moved. The robot system according to claim 9.
11. The aforementioned visual sensor is a sensor capable of acquiring two-dimensional images and three-dimensional information. The robot system according to claim 9.
12. A control method that receives the output of a visual sensor that images a moving workpiece and controls the processing of the workpiece, The steps include acquiring a two-dimensional image including the workpiece based on the output of the visual sensor, The steps include generating three-dimensional information including the workpiece based on the output of the visual sensor, A step of limiting the processing range in the acquired two-dimensional image, The process includes the step of changing the relative position of the three-dimensional information based on a limited processing range of the two-dimensional image in a predetermined coordinate system. Control method.