Control device, machine system, and control program

Abstract

Provided is a control device that can efficiently adjust an image processing parameter. A control device according to the present invention comprises a queue management unit that manages the positions of each of a plurality of workpieces on the basis of the output of an image processing device that processes images of the workpieces that have been captured by a visual sensor and compares an image processing score for each workpiece with a threshold value to determine detection of the workpiece and the output of a position detector that detects the position of each workpiece as transported by a transport device. The queue management unit acquires information about first workpieces for which the image processing score is at or above a first threshold value, information about second workpieces for which the image processing score is below the first threshold value but at or above a second threshold value, and information about second workpiece images that have been captured of the second workpieces and specifies images to be used to improve the image processing on the basis of the output of the position detector and the information about the second workpieces.

Description

Control device, mechanical system and control program

[0001] The present disclosure relates to a control device, a machine system, and a control program.

[0002] In recent years, robot systems have been put into practical use that use visual sensors to capture images of objects (workpieces) being transported by conveyors or other transport devices, process the captured images to detect the workpieces, and have a robot perform tasks such as picking up the workpieces based on the detected position of the workpieces.

[0003] In such a robot system, if the robot fails to pick up a workpiece from the conveyor, the image processing parameters are adjusted using an image of the failed pick-up (failed image).That is, the accuracy of workpiece detection is improved by checking whether the workpiece was not detected from the history of captured images and adjusting the image processing parameters using the image in the case where the workpiece was not detected.

[0004] Conventionally, various proposals have been made to improve the storage process of captured images in robot control devices (control devices) that capture images of workpieces being transported by a transport device and cause a robot to perform specified processing on the workpiece based on those images.

[0005] Japanese Patent Laid-Open No. 09-131683 Japanese Patent Laid-Open No. 2020-032520 Japanese Patent Laid-Open No. 2012-166308

[0006] As mentioned above, a robot system has been put into practical use in which a visual sensor captures images of workpieces being transported by a conveyor or the like, and the robot then picks up the workpieces. However, there are a large number of images of robots failing to pick up a workpiece from the conveyor, and adjusting image processing parameters using these images would result in an enormous amount of work.

[0007] Note that image processing parameter adjustment is not limited to the robot system that picks up the workpiece as described above, but is also applicable to various robot systems (machine systems) that have a robot (machine) perform specified processing or machining on the workpiece.

[0008] Therefore, there is a demand for a control device, a machine system, and a control program that can efficiently adjust parameters for image processing.

[0009] According to one embodiment of the present disclosure, there is provided a control device that controls a machine to perform specified operations on multiple workpieces transported by a transport device, and that includes a queue management unit that manages the position of each workpiece based on the output of an image processing device that processes images of multiple workpieces captured by a visual sensor and compares the image processing score for each workpiece with a threshold value to determine whether the workpiece has been detected, and the output of a position detector that detects the position of each workpiece transported by the transport device.

[0010] The queue management unit acquires information on a first workpiece whose image processing score is equal to or greater than a first threshold, information on a second workpiece whose image processing score is less than the first threshold but equal to or greater than a second threshold, and information on an image of the second workpiece in which the second workpiece is captured, and identifies an image to be used to improve the image processing based on the output of the position detector and the information on the second workpiece.

[0011] Fig. 1 is a front view schematically showing the overall configuration of an example of a robot system according to this embodiment. Fig. 2 is a plan view of the robot system shown in Fig. 1. Fig. 3 is a functional block diagram showing an example of a robot control device (control device) in the robot system shown in Figs. 1 and 2. Fig. 4 is a flowchart for explaining an example of processing in an example of a control program according to this embodiment. Fig. 5 is a flowchart for explaining another example of processing in an example of a control program according to this embodiment.

[0012] Below, examples of a control device, a machine system, and a control program according to the present embodiment will be described in detail with reference to the accompanying drawings. In each drawing, identical or similar components are assigned identical or similar reference numerals. Furthermore, the embodiments described below do not limit the technical scope and meaning of the terms of the invention described in the claims. Note that, in this specification, the term "machine" refers to various machines, including various robots such as industrial robots and collaborative robots. Furthermore, in this specification, the processing performed by a robot is not limited to the picking of workpieces moving on a conveyor, but also includes various machining and processing performed on moving workpieces.

[0013] Fig. 1 is a front view showing a schematic diagram of the overall configuration of one example of a robot system according to this embodiment, and Fig. 2 is a plan view of the robot system shown in Fig. 1. In Fig. 1 and Fig. 2, reference numeral 100 denotes a robot system (mechanical system), 1 denotes a robot (machine), 2 denotes a control device (robot control device), 3 denotes an imaging device, 4 denotes an image processing device, 5 denotes a conveyor position detection device, 6 denotes a conveyor, and W denotes a workpiece. Note that in Fig. 1 and Fig. 2, the image processing device 4 is built into the robot control device 2, but it can also be provided independently from the robot control device 2.

[0014] As shown in FIGS. 1 and 2 , the robot system 100 includes a robot 1 equipped with a hand 10, a robot control device 2 incorporating an image processing device 4, an imaging device 3, and a conveyor 6. The robot 1 includes a base 14 fixed to an installation surface, a swivel base 13 that rotates relative to the base 14, a lower arm 12 rotatably supported on the swivel base 13, an upper arm 11 rotatably supported on the lower arm 12, and a wrist 15 rotatably supported at the end of the upper arm 11. The upper arm 11 rotates around an axis of rotation parallel to the direction of extension of the upper arm 11, and a rotatably formed flange 16 is provided at the tip of the wrist 15. In the description of this embodiment, the robot 1 is an articulated robot having multiple joints, but this is not limited to this configuration. Any robot equipped with various work tools can be used. Furthermore, while the workpiece W is illustrated as a rectangular parallelepiped cardboard box as an example, the workpiece W is not limited to a cardboard box and may be made of various materials and shapes.

[0015] The hand 10 is a work tool capable of gripping a workpiece W, and is configured to, for example, suck and grip the workpiece W using a plurality of suction pads 2a. The hand 10 is fixed to a flange 16 of a wrist 15, but any work tool can be applied as the work tool attached to the robot 1.

[0016] The conveyor 6 is an example of a transport device that transports the workpiece W, and transports the workpiece W in a predetermined direction by rotating a circular belt 6a with a drive motor M. That is, as shown by the arrow TD, the workpiece W is moved horizontally, and the robot 1 changes its position and posture to transport the workpiece W to a position where the hand 10 can grasp the workpiece W. The drive motor M is provided with an encoder that functions as a conveyor position detector 5 that detects the position of the workpiece W. Note that the conveyor position detector 5 is not limited to an encoder provided in the drive motor M; for example, it may be an encoder that directly calculates the movement of the belt 6a.

[0017] The imaging device 3 includes, for example, a visual sensor (camera) 30 that captures at least one of a two-dimensional image and a three-dimensional image of the workpiece W, and the image captured by the visual sensor 30 is output to the image processing device 4 (robot control device 2). Here, the imaging device 3 may be a two-dimensional camera having an imaging range (field of view) FV, but various known visual sensors can also be applied, such as a three-dimensional camera equipped with two cameras and one imaging pattern light-emitting unit.

[0018] The visual sensor 30 is supported by a support member 31 and is positioned so as to capture an image of the workpiece W being transported by the conveyor 6. The visual sensor 30 is positioned upstream of the robot 1 in the direction TD in which the workpiece W is transported. The robot system 100 is provided with a world coordinate system WC as a reference coordinate system, and the origin of the world coordinate system WC is set, for example, on the base 14 of the robot 1. This ensures that the position and orientation of the world coordinate system WC do not change even if the position and posture of the robot 1 change. The world coordinate system WC has X-, Y-, and Z-axes that are orthogonal to each other as coordinate axes, and W-, P-, and R-axes are set as coordinate axes around the X-, Y-, and Z-axes.

[0019] Furthermore, the robot system 100 is provided with a tool coordinate system TC having an origin set at an arbitrary position on the work tool, and the origin of the tool coordinate system TC is set at the tool center point of the hand 10. As a result, when the position and orientation of the robot 1 change, the position and orientation of the tool coordinate system TC also change. For example, the position of the robot 1 corresponds to the position of the tool center point, and the orientation of the robot 1 corresponds to the orientation of the tool coordinate system TC relative to the world coordinate system WC. Furthermore, the robot system 100 is provided with a sensor coordinate system SC corresponding to the visual sensor 30, and the origin of the sensor coordinate system SC is fixed to the visual sensor 30. As a result, coordinate values ​​in the sensor coordinate system SC can be converted to coordinate values ​​in the world coordinate system WC based on the position and orientation of the sensor coordinate system SC relative to the world coordinate system WC.

[0020] Figure 3 is a functional block diagram showing an example of a robot control device (control device) in the robot system shown in Figures 1 and 2. Here, in the robot system 100 shown in Figures 1 and 2, the robot control device 2 has a built-in image processing device 4, but in Figure 3, the robot control device 2 and the image processing device 4 are depicted as separate entities. That is, the image processing device 4 may be integrally built into the robot control device 2, or it may be provided externally to the robot control device 2 as a separate entity.

[0021] In FIG. 3 , reference numerals 1, 2, 4, 5, and 30 denote a robot, a robot control device, an image processing device, a conveyor position detector, and a visual sensor. As shown in FIG. 3 , the robot control device 2 controls the robot 1 to perform a predetermined task, such as picking up a workpiece W transported by a conveyor 6, and includes an operation program 21, an image processing trigger unit 22, a queue management unit 23, a queue 24, an operation control unit 25, and a robot drive unit 26. The image processing device 4 processes images of multiple workpieces W captured by the visual sensor 30 and compares the image processing score for each workpiece W with a threshold value to determine whether that workpiece W has been detected. The image processing device 4 includes a detection unit 41, a result management unit 42, an image buffer 43, and an image storage unit 44. The image buffer 43 and the image storage unit 44 constitute a storage device for storing (retaining) image data, etc.

[0022] The operation program 21 is for controlling the robot 1 to perform predetermined tasks on the workpieces W transported by the conveyor 6, and is stored in a storage unit such as a ROM (Read Only Memory) or flash memory, and is executed by a microprocessor (MPU) or the like that functions as an arithmetic processing unit. The arithmetic processing unit is also used to execute the functions of the queue management unit 23 and the operation control unit 25, etc. The image processing trigger unit 22 receives the output of the conveyor position detector 5 that detects the position of each workpiece W transported by the conveyor 6, and issues a trigger to the image processing unit 4, and the image processing unit 4 issues an image trigger to the visual sensor 30.

[0023] The queue management unit 23 receives the output of the image processing device 4 (result management unit 42) and the output of the conveyor position detector 5, and manages the position of each workpiece W. That is, the queue management unit 23 receives from the image processing device 4 information on detected successfully works (first works) W whose image processing scores are equal to or greater than a first threshold, information on specific detected unsuccessful works (second works) W whose image processing scores are less than the first threshold and equal to or greater than a second threshold, and information on specific detected unsuccessful images (second workpiece images) in which the specific detected unsuccessful works W are captured.

[0024] Furthermore, the queue management unit 23 identifies images to be used to improve image processing in the image processing device 4 based on the output of the conveyor position detector 5 and information on the identified detection-failed workpieces W. The queue 24 receives the output of the queue management unit 23 and stores information on the detected workpieces W, information on the identified detection-failed workpieces W, and information on the images for which detection has been performed. The operation control unit 25 generates operation commands to control the operation of the robot 1 based on the operation program 21 and the output of the queue management unit 23. The robot driving unit 26 drives and controls the robot 1 based on the operation commands output from the operation control unit 25.

[0025] The detection unit 41 receives images of multiple workpieces W captured by the visual sensor 30, compares the image processing score for each workpiece W with a threshold value to detect (determine), and outputs the determination result to the result management unit 42. Here, the detection unit 41 prepares two threshold values, a first threshold value (e.g., 90%) and a second threshold value (e.g., 80%), for determining the image processing score for the workpiece W (e.g., template matching similarity: %). The result management unit 42 receives the results of the comparison and detection by the detection unit 41 with the first and second threshold values, and outputs to the queue management unit 23 information on workpieces W that were successfully detected, information on workpieces W that failed to be identified for detection, and information on images where identification for detection failed.

[0026] Specifically, if the image processing score for the workpiece W is equal to or greater than a first threshold value (90%), the image data for that workpiece W is determined to be detection-successful image data, and if the image processing score for the workpiece W is less than the first threshold value (90%), the image data for that workpiece W is determined to be detection-failed image data. Here, the robot control device (image processing device) according to this embodiment compares and detects the image processing score for the workpiece W with a second threshold value (80%) that is smaller than the first threshold value (90%).

[0027] That is, if the image processing score for the work W is less than the first threshold (90%) and equal to or greater than the second threshold (80%), the image data of that work W is determined to be specific detection failed image data. Note that if the image processing score for the work W is less than the second threshold (80%), the image data of that work W is determined to be detection failed image data as is. Here, specific detection failed image data of the work W for which the image processing score for the work W is less than the first threshold (90%) and equal to or greater than the second threshold (80%) can be used, for example, to adjust the parameters of the image processing device 4, thereby making it possible to improve the detection accuracy of the work W.

[0028] Setting the first threshold value to 90% and the second threshold value to 80% described above is merely an example, and the values ​​of the first threshold value and the second threshold value vary depending on, for example, the type of work or processing to be performed on the workpiece W to which the robot system 100 is applied. That is, the first threshold value and the second threshold value are set to values ​​that enable, for example, parameter adjustment of the image processing device 4 to be performed effectively (efficiently) in a short time when detection failure image data (specific detection failure image data) that is less than the first threshold value but equal to or greater than the second threshold value is used.

[0029] Here, the specific detection failure image data that is less than the first threshold value but equal to or greater than the second threshold value is input to a development computer installed, for example, in an office or the like remote from the factory where the robot 1, robot control device 2, etc. are installed, and the parameters of the image processing device 4 are adjusted to improve the detection accuracy of the workpiece W. Note that the image data from the image storage unit 44 and the queue 24, size information, position information, etc. of the workpiece W can be input to the development computer via storage means such as a USB memory or wired or wireless communication means, for example.

[0030] First, in the robot control device 2 (image processing device 4) of the first example according to this embodiment, the storage device is composed of an image buffer 43 and an image storage unit 44, and stores specific detection failure image data in which at least specific detection failure workpieces W are imaged, based on the output of the result management unit 42. That is, in the first example according to this embodiment, the image buffer 43 holds image data in which each workpiece W is imaged, and the image storage unit 44 stores only specific detection failure image data in which specific detection failure workpieces W are imaged for which the image processing score is less than the first threshold value and equal to or greater than the second threshold value.

[0031] Furthermore, in the robot control device 2 of the first embodiment, the queue management unit 23 writes the position of each workpiece W into the queue 24 together with the value of the conveyor position detector 5 when the image of that workpiece W was acquired. Furthermore, when storing a workpiece W in the queue 24, the queue management unit 23 determines whether the value of the conveyor position detector 5 is within the tolerance range when it is the same as that of a successfully detected workpiece (first workpiece) W in the queue 24, and if it determines that it is within the tolerance range, it determines that it is the same workpiece W and discards one of the results. Furthermore, if there is a specific workpiece (second workpiece) W for which detection has failed, the queue management unit 23 stores the specific unsuccessful detection image data (second workpiece image data) in the image buffer 43 and writes it into the queue 24 as a unsuccessful detection workpiece W.

[0032] The queue management unit 23 determines whether or not there is a workpiece W identical to the newly stored successfully detected workpiece W among the unsuccessfully detected workpieces W in the queue 24, and if it determines that there is, discards the unsuccessfully detected workpieces W in the queue 24, and further discards the image data held in the image buffer 43 when all of the unsuccessfully detected workpieces W in the queue 24 have been discarded. Then, based on the size information of each workpiece W and the current position calculated from the value of the conveyor position detector 5, the queue management unit 23 stores the image data held in the image buffer 43 when the workpiece W is completely out of the field of view (outside the imaging range of the visual sensor 30) in the image storage unit 44 as specific unsuccessfully detected image data, and identifies it as an image to be used to improve image processing in the image processing device 4.

[0033] Here, the image processing device 4 performs template matching, for example, by comparing the shape of each workpiece W captured by the visual sensor 30 with previously prepared shape data of the workpiece W. The similarity (score) between the two is calculated and compared with thresholds (first and second thresholds). However, the comparison of the image processing score and the threshold is not limited to the score (similarity) obtained by template matching; various image processing scores can be compared with the thresholds. Thus, according to the robot control device 2 (image processing device 4) of the first embodiment, the image storage unit 44 only needs to store image data (specific detection failure image data) used for parameter adjustment of the image processing device 4, thereby significantly reducing the memory capacity of the storage device (image storage unit 44). The parameter adjustment of the image processing device can be based on, for example, the output of the conveyor position detector 5, information on the specific detection failure workpiece W, and images identified by the robot control device 2.

[0034] Next, in the robot control device 2 (image processing device 4) of the second example according to this embodiment, the storage device is configured with an image storage unit 44 that stores all image data obtained by capturing images of each workpiece W. That is, in the second example according to this embodiment, the image storage unit 44 stores all image data including specific detection failure image data obtained by capturing images of specific detection failure workpieces W whose image processing scores are less than the first threshold value and equal to or greater than the second threshold value. Note that in this second example, the image buffer 43 in the first example described above does not need to be provided as a storage device.

[0035] As described above, in the second embodiment, the queue management unit 23 writes the position of each workpiece W in the queue 24 together with the value of the conveyor position detector 5 when the image of that workpiece W was acquired, and stores all image data of each workpiece W in the image storage unit 44. Here, all image data of each workpiece W stored in the image storage unit 44 is determined based on the value of the conveyor position detector 5 so that the range imaged by the visual sensor 30 is continuous.

[0036] That is, the image processing trigger unit 22 recognizes that the conveyor 6 has moved a certain distance (the imaging range of the visual sensor 30) based on the output of the conveyor position detector 5, outputs a trigger signal, and stores all image data in the image storage unit 44 so that images captured for each imaging range (field of view) of the visual sensor 30 are continuous. As a result, the image storage unit 44 stores not only specific detection failure image data, but also detection success image data and all detection failure image data, requiring a large memory capacity. All image data stored in the image storage unit 44 is input, for example, via a USB memory or communication means, to a development computer installed in an office or other location away from the factory, where data to be used for parameter adjustment of the image processing device 4 is selected.

[0037] Furthermore, in the user interface that displays historical images in chronological order from oldest to newest, if there is a workpiece W for which detection has failed, the queue management unit 23 identifies the image containing that workpiece W based on the size information of each workpiece W and the current position calculated from the value of the conveyor position detector 5, and also performs comparative detection on the identified image. Then, based on the value of the conveyor position detector 5 corresponding to the image data stored in the image storage unit 44, if there is not a single workpiece W determined to be the same in the image data for which detection has been successful, the queue management unit 23 identifies these image data as images to be used to improve image processing in the image processing device 4.

[0038] As described above, in the first and second examples of the control device (robot control device) according to this embodiment, the image processing score can be determined based on the similarity of the template matching of the workpiece in the captured image, but is not limited to the similarity of the template matching of the workpiece, and various known image processing scores can be applied.

[0039] 4 is a flowchart illustrating an example of processing in an embodiment of the control program according to this embodiment, and is intended to illustrate the control processing during workpiece detection in the first embodiment of this embodiment. As shown in FIG. 4, when the control processing during workpiece detection in the embodiment of the control program according to this embodiment starts (START), in step ST11, the image processing trigger unit 22 triggers the visual sensor 30 to capture an image (the image processing trigger unit 22 triggers the image processing device 4, and the image processing device 4 then triggers the visual sensor 30), and the process proceeds to step ST12. As described above, the detection unit 41 of the image processing device 4 receives images of multiple workpieces W captured by the visual sensor 30 and compares the image processing score for each workpiece W with a threshold value.

[0040] That is, in step ST12, the detection unit 41 acquires an image from the visual sensor 30, compares and detects (determines) the image processing score for each work W with the first threshold value and the second threshold value, and outputs the determination result to the result management unit 42. Here, as described above, in the first example according to this embodiment, a detection-failed work (specific detection-failed work) that is less than the first threshold value and equal to or greater than the second threshold value is detected, and the process proceeds to step ST13.

[0041] In step ST13, the result management unit 42 of the image processing device 4: adds an ID (identification number) and stores the image (image data) in the image buffer 43; and transmits the successful detection work information, the unsuccessful detection work information, and the image ID to the queue management unit 23 of the robot control device 2. As described above, the successful detection work information is information on successful detection work when the image processing score for the work W is equal to or greater than the first threshold, and the unsuccessful detection work information is information on unsuccessful detection work when the image processing score for the work W is less than the first threshold. It goes without saying that a specific unsuccessful detection work whose image processing score is less than the first threshold but equal to or greater than the second threshold is included in the unsuccessful detection work whose image processing score is less than the first threshold.

[0042] Next, proceed to step ST14, where the queue management unit 23 stores in the queue 24 only the successfully detected works that do not have any successfully detected works that can be considered to be the same work, stores the image ID in the queue 24, stores the unsuccessfully detected works in the queue 24 linked to the image ID, deletes from the queue 24 the unsuccessfully detected works that have successfully detected works that can be considered to be the same work, and notifies the result management unit 42 of the image processing device 4 of the IDs from which all linked unsuccessful detected works have been deleted.

[0043] Furthermore, the process proceeds to step ST15, where the result management unit 42 of the image processing device 4 deletes the image of the notified ID from the image buffer 43, and ends (END) the control processing at the time of workpiece detection in one example of the control program according to the embodiment. Here, the processing in one example of the control program according to the embodiment described with reference to Figure 4 stores image data in the image storage unit 44 only image data of detection-failed works that are suitable for efficiently adjusting the image processing parameters, that is, image data of specific detection-failed works whose image processing score is less than the first threshold and equal to or greater than the second threshold, thereby making it possible to significantly reduce the memory capacity of the image storage unit 44.

[0044] 5 is a flowchart illustrating another example of processing in an embodiment of the control program according to this embodiment, and is intended to illustrate the control processing for determining the current position of the workpiece. As shown in FIG. 5, when the control processing for determining the current position of the workpiece in an embodiment of the control program according to this embodiment starts (START), in step ST21, the queue management unit 23 determines whether the entire workpiece W is outside the field of view of the visual sensor 30 based on the current position and workpiece outer shape information of the workpiece that has failed detection in the queue 24. That is, if it is determined in step ST22 that the entire workpiece W is not outside the field of view of the visual sensor 30 (NO), the control processing for determining the current position of the workpiece is ended (END).

[0045] On the other hand, if it is determined in step ST22 that the entire work W is outside the field of view of the visual sensor 30 (YES), the process proceeds to step ST23, where the image ID linked to the corresponding detection-failed work W is deleted from the queue 24, and the deleted image ID is notified to the result management unit 42 of the image processing device 4. Here, the detection-failed work W is a work W for which the image processing score for that work W is less than the first threshold value, and if the entire work W is outside the field of view (outside the imaging range) of the visual sensor 30, the image ID linked to that detection-failed work W is deleted from the queue 24, and the image data corresponding to the deleted ID is notified to the result management unit 42.

[0046] Furthermore, the process proceeds to step ST24, where the result management unit 42 of the image processing device 4 stores the image (image data) corresponding to the ID in the image storage unit 44, deletes it from the image buffer 43, and ends (END) the control process for determining the current position of the workpiece. Note that the process of one example of the control program according to this embodiment described with reference to Figures 4 and 5 is merely an example, and it goes without saying that various changes and modifications are possible.

[0047] The control program according to the present embodiment described above may be provided by recording it on a computer-readable non-transitory recording medium or non-volatile semiconductor memory, or may be provided via a wired or wireless connection. Examples of the computer-readable non-transitory recording medium include optical disks such as CD-ROMs (Compact Disc Read Only Memory) and DVD-ROMs, or hard disk drives. Examples of the non-volatile semiconductor memory include PROMs (Programmable Read Only Memory) and flash memory. Furthermore, the program may be distributed from a server device via a wired or wireless LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0048] As described above in detail, the control device, machine system, and control program according to this embodiment make it possible to efficiently adjust parameters for image processing.

[0049] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0050] The following supplementary notes are further disclosed regarding the above-described embodiment and modified examples. [Supplementary Note 1] A control device (2) controls a machine (1) to perform a predetermined operation on a plurality of workpieces (W) transported by a transport device (6), and includes a queue management unit (23) that manages the position of each of the workpieces (W) based on the output of an image processing device (4) that processes images of the plurality of workpieces (W) captured by a visual sensor (30) and compares an image processing score for each of the workpieces (W) with a threshold value to determine detection of the workpiece (W), and the output of a position detector (5) that detects the position of each of the workpieces (W) transported by the transport device (6), wherein the queue management unit (23) acquires information on a first workpiece (W) whose image processing score is equal to or greater than a first threshold value, information on a second workpiece (W) whose image processing score is less than the first threshold value but equal to or greater than a second threshold value, and information on a second workpiece image obtained by capturing an image of the second workpiece (W), and the control device identifies an image to be used for improving the image processing based on the output of the position detector (5) and the information on the second workpiece (W). [Supplementary Note 2] The control device according to Supplementary Note 1, further comprising an image processing trigger unit (22) that receives an output from the position detector (5) and triggers the image processing device (4), and the image processing device (4) issues an image trigger to the visual sensor (30). [Supplementary Note 3] The control device according to Supplementary Note 1 or Supplementary Note 2, wherein the control device (2) has the image processing device (4) built in.[Supplementary Note 4] The image processing device (4) has: a detection unit (41) that receives images of the plurality of workpieces (W) captured by the visual sensor (30) and compares and detects the image processing score for each of the workpieces (W) with the first threshold value and the second threshold value; a result management unit (42) that receives the result of the comparison and detection with the first threshold value and the second threshold value by the detection unit (41) and outputs information about the first workpiece (W), information about the second workpiece (W), and information about the second workpiece image to the control device (4) for the queue management unit (23); and a storage device (43, 44) that stores second workpiece image data of at least the second workpiece (W) based on the output of the result management unit (42). [Appendix 5] The control device according to Appendix 4, wherein the storage device comprises: an image buffer (43) for holding image data of each of the workpieces (W); and an image storage unit (44) for storing only the second workpiece image data of the second workpiece (W) whose image processing score is less than the first threshold value and greater than or equal to the second threshold value.[Supplementary Note 6] The queue management unit (23) writes the position of each of the works (W) into a queue (24) together with the value of the position detector (5) when the image of the work (W) was acquired, determines whether the value of the position detector (5) is within an allowable range when storing the work (W) in the queue (24) when the value is the same as that of the first work (W) in the queue (24), and if it is determined that the value is within the allowable range, determines that the work (W) is the same as the first work (W) and discards one of the results, if the second work (W) is present, holds the second work image data in the image buffer (43) and writes it into the queue (24) as a work (W) that has failed to be detected, determines whether a work (W) that is the same as the newly stored first work (W) is present among the work (W) that has failed to be detected in the queue (24), and if it is determined that the work (W) is present, discards the work (W) that has failed to be detected in the queue (24), The control device according to Supplementary Note 5, wherein when all of the detection-failed workpieces (W) in the queue (24) have been discarded, the image data held in the image buffer (43) is discarded, and when the workpiece (W) has completely gone out of the field of view, the image data held in the image buffer (43) is stored in the image storage unit (44) as the second workpiece image data based on size information of each of the workpieces (W) and a current position calculated from the value of the position detector (5), and the image is identified as an image to be used for improving image processing. [Supplementary Note 7] The control device according to Supplementary Note 5, wherein the storage device comprises an image storage unit (44) for storing all image data obtained by imaging each of the workpieces (W), and all the image data includes the second workpiece image data obtained by imaging the second workpiece (W) whose score in the image processing is less than the first threshold value and equal to or greater than the second threshold value.[Appendix 8] The queue management unit (23) writes the position of each of the workpieces (W) into the queue (24) together with the value of the position detector (5) when the image of the workpiece (W) was acquired, and stores all image data of each of the workpieces (W) in the image storage unit (44), and in a user interface that displays historical images in chronological order from oldest to newest, if there is a second workpiece (W), identifies an image containing the workpiece (W) based on size information of each of the workpieces (W) and a current position calculated from the value of the position detector (5), and also performs the comparative detection on the identified image, and when there is no workpiece (W) in the successfully detected image data that is determined to be the same based on the value of the position detector (5) corresponding to the image data stored in the image storage unit (44), identifies these image data as images to be used for improving image processing in the image processing device (4). This is the control device described in Appendix 7. [Supplementary Note 9] The control device according to Supplementary Note 8, wherein all image data of each of the workpieces (W) stored in the image storage unit (44) are determined based on the value of the position detector (5) so that the range imaged by the visual sensor (30) is continuous. [Supplementary Note 10] The control device according to any one of Supplementary Notes 1 to 9, wherein the machine is a robot (1), the transport device (6) is a conveyor, the control device is a robot control device (2) that controls the robot (1) to perform a predetermined task for each of the workpieces (W) transported by the conveyor based on an operation program (21), and the position detector is a conveyor position detector (5) that detects the position of the workpieces (W) moving on the conveyor.[Supplementary Note 11] The control device according to Supplementary Note 10, further comprising: a queue (24) that stores information on the first workpiece (W), information on the second workpiece (W), and information identifying the image in which the detection was performed based on the output of the queue management unit (23); an operation control unit (25) that generates an operation command to control the operation of the robot (1) based on the operation program (21) and the output of the queue management unit (23); and a robot driving unit (26) that drives the robot (1) based on the operation command output from the operation control unit. [Supplementary Note 12] The control device according to any one of Supplements 1 to 11, wherein the image processing score is determined based on a similarity of template matching of the workpiece (W) in an image captured by the visual sensor (30). [Supplementary Note 13] The control device according to any one of Supplements 1 to 12, wherein parameters in the image processing device (4) are adjusted based on the output of the position detector (5), information on the second workpiece (W), and the image identified by the control device (2). [Supplementary Note 14] A machine system (100) comprising: a machine (1); a control device (2) that controls the machine (1) to perform predetermined operations on a plurality of workpieces (W) transported by a transport device (6); and an image processing device (4) that processes images of the plurality of workpieces (W) captured by a visual sensor (30) and compares an image processing score for each of the workpieces (W) with a threshold value to determine detection of the workpieces (W), wherein the control device (2) comprises a queue management unit (23) that manages the position of each of the workpieces (W) based on an output of the image processing device (4) and an output of a position detector (5) that detects the position of each of the workpieces (W) transported by the transport device (6), and the queue management unit (23) acquires information on a first workpiece (W) whose image processing score is equal to or greater than a first threshold value, information on a second workpiece (W) whose image processing score is less than the first threshold value and equal to or greater than a second threshold value, and information on a second workpiece image in which the second workpiece (W) is captured, A machine system that identifies an image to be used for improving image processing based on the output of the position detector (5) and information about the second workpiece (W).[Supplementary Note 15] In a machine system (100) comprising a machine (1), a control device (2) that controls the machine (1) to perform predetermined operations on a plurality of workpieces (W) transported by a transport device (6), and an image processing device (4) that processes images of the plurality of workpieces (W) captured by a visual sensor (30) and compares an image processing score for each of the workpieces (W) with a threshold value to determine detection of the workpieces (W), the control program specifies images to be used to improve image processing, the control program causing an arithmetic processing device to execute the steps of: managing the position of each of the workpieces (W) based on the output of the image processing device (4) and the output of a position detector (5) that detects the position of each of the workpieces (W) transported by the transport device (6); and acquiring information on a second workpiece (W) whose image processing score is less than the first threshold value and is greater than or equal to a second threshold value, in addition to information on a first workpiece (W) whose image processing score is greater than or equal to a first threshold value.

[0051] REFERENCE SIGNS LIST 1 Robot (machine) 2 Robot control device (control device) 3 Imaging device 4 Image processing device 5 Conveyor position detector (position detector) 6 Conveyor (transport device) 21 Operation program 22 Image processing trigger unit 23 Queue management unit 24 Queue 25 Operation control unit 26 Robot driving unit 30 Visual sensor 41 Detection unit 42 Result management unit 43 Image buffer 44 Image storage unit 100 Robot system (mechanical system) FV Imaging range (field of view) SC Sensor coordinate system TC Tool coordinate system W Work (object) WC World coordinate system

Claims

1. A control device that controls a machine to perform specified tasks on multiple workpieces transported by a transport device, and includes a queue management unit that processes images of the multiple workpieces captured by a visual sensor and manages the position of each of the workpieces based on the output of an image processing device that compares the image processing score for each of the workpieces with a threshold value to determine whether the workpiece has been detected, and the output of a position detector that detects the position of each of the workpieces transported by the transport device, wherein the queue management unit obtains information on first workpieces whose image processing score is equal to or greater than a first threshold value, information on second workpieces whose image processing score is less than the first threshold value but equal to or greater than a second threshold value, and information on second workpiece images of the second workpieces, and identifies images to be used to improve image processing based on the output of the position detector and the information on the second workpieces.

2. The control device according to claim 1, further comprising an image processing trigger unit that receives the output of the position detector and issues a trigger to the image processing device, and the image processing device issues an image trigger to the visual sensor.

3. The control device according to claim 1 or claim 2, wherein the control device has the image processing device built in.

4. The image processing device according to any one of claims 1 to 3, further comprising: a detection unit that receives images of the plurality of workpieces captured by the visual sensor and compares and detects the image processing score for each of the workpieces with the first threshold value and the second threshold value; a result management unit that receives the results of the comparison and detection by the detection unit with the first threshold value and the second threshold value and outputs information on the first workpiece, information on the second workpiece, and information on the second workpiece image to the control device for the queue management unit; and a storage device that stores second workpiece image data of at least the second workpiece based on the output of the result management unit.

5. The control device described in claim 4, wherein the memory device comprises: an image buffer that holds image data of each of the workpieces; and an image storage unit that stores only the second workpiece image data of the second workpiece whose image processing score is less than the first threshold value and greater than or equal to the second threshold value.

6. The control device according to claim 5, wherein the queue management unit writes the position of each of the workpieces into the queue together with the value of the position detector when the image of the workpiece was acquired, determines whether the value of the position detector is the same as that of the first workpiece in the queue when storing the workpiece in the queue is within an acceptable range, and if it is determined that it is within the acceptable range, determines that the works are the same and discards one of the results, if the second workpiece is present, holds the image data of the second workpiece in the image buffer and writes it into the queue as a workpiece that has failed to be detected, determines whether a workpiece that is the same as the newly stored first workpiece is among the works that have failed to be detected in the queue, and if it is determined that there is a workpiece that is the same as the first workpiece, discards the workpiece that has failed to be detected in the queue, discards the image data held in the image buffer when all of the works that have failed to be detected in the queue have been discarded, and stores the image data held in the image buffer when the workpiece has completely gone out of the field of view based on the size information of each of the works and the current position calculated from the value of the position detector in the image storage unit, and identifies it as an image to be used for improving image processing.

7. The control device described in claim 5, wherein the memory device is provided with an image storage unit that stores all image data of each of the workpieces, and all of the image data includes the second workpiece image data of the second workpiece whose image processing score is less than the first threshold value and greater than or equal to the second threshold value.

8. The control device according to claim 7, wherein the queue management unit writes the position of each of the workpieces into a queue together with the value of the position detector when the image of the workpiece was acquired, and stores all image data of each of the workpieces in the image storage unit; in a user interface that displays historical images in chronological order from oldest to newest, if there is a second workpiece, identifies an image containing that workpiece based on the size information of each of the workpieces and the current position calculated from the value of the position detector, and also performs the comparative detection on the identified image; and when there is no workpiece in the successfully detected image data that is determined to be the same based on the value of the position detector corresponding to the image data stored in the image storage unit, identifies these image data as images to be used to improve image processing in the image processing device.

9. A control device as described in claim 8, wherein all image data of each of the workpieces stored in the image storage unit is determined based on the value of the position detector so that the range imaged by the visual sensor is continuous.

10. A control device according to any one of claims 1 to 9, wherein the machine is a robot, the transport device is a conveyor, the control device is a robot control device that controls the robot to perform a specified task on each of the workpieces transported by the conveyor based on an operating program, and the position detector is a conveyor position detector that detects the position of the workpieces moving on the conveyor.

11. The control device described in claim 10, further comprising: a queue that stores information about the first workpiece, information about the second workpiece, and information identifying the image on which the detection was performed based on the output of the queue management unit; an operation control unit that generates operation commands that control the operation of the robot based on the operation program and the output of the queue management unit; and a robot driving unit that drives the robot based on the operation commands output from the operation control unit.

12. A control device according to any one of claims 1 to 11, wherein the image processing score is determined based on the degree of similarity of template matching of the workpiece in the image captured by the visual sensor.

13. A control device according to any one of claims 1 to 12, which adjusts parameters in the image processing device based on the output of the position detector, information on the second workpiece, and the image identified by the control device.

14. A machine system comprising: a control device that controls the machine to perform specified operations on multiple workpieces transported by a transport device; and an image processing device that processes images of the multiple workpieces captured by a visual sensor and compares the image processing score for each of the workpieces with a threshold value to determine whether the workpiece has been detected, wherein the control device has a queue management unit that manages the position of each of the workpieces based on the output of the image processing device and the output of a position detector that detects the position of each of the workpieces transported by the transport device, and the queue management unit acquires, in addition to information on first workpieces whose image processing score is equal to or greater than a first threshold value, information on second workpieces whose image processing score is less than the first threshold value but equal to or greater than a second threshold value, and information on second workpiece images obtained by capturing images of the second workpieces, and identifies images to be used to improve image processing based on the output of the position detector and the information on the second workpieces.

15. A control program for specifying images to be used to improve image processing in a machine system comprising a machine, a control device that controls the machine to perform specified tasks on multiple workpieces transported by a transport device, and an image processing device that processes images of the multiple workpieces captured by a visual sensor and compares the image processing score for each of the workpieces with a threshold value to determine whether the workpiece has been detected, the control program causing an arithmetic processing device to execute the following steps: managing the position of each of the workpieces based on the output of the image processing device and the output of a position detector that detects the position of each of the workpieces transported by the transport device; and acquiring information about a first workpiece whose image processing score is equal to or greater than a first threshold value, information about a second workpiece whose image processing score is less than the first threshold value but equal to or greater than a second threshold value, and information about a second workpiece image in which the second workpiece is captured.

Images