Control device, mechanical system, and control program

Abstract

Provided is a control device capable of efficiently performing parameter adjustment of image processing. The control device includes a queue management unit that manages positions of individual workpieces based on an output of an image processing device and an output of a position detector, wherein the image processing device processes images of a plurality of workpieces captured by a vision sensor and determines detection of the workpieces by comparing a score of image processing for each workpiece with a threshold value, and the position detector detects positions of individual workpieces conveyed by a conveyance device. The queue management unit acquires information of a first workpiece whose score of image processing is equal to or higher than a first threshold value, and acquires information of a second workpiece whose score of image processing is lower than the first threshold value and equal to or higher than a second threshold value, and information of a second workpiece image captured by capturing the second workpiece, and determines an image used in improvement of image processing based on the output of the position detector and the information of the second workpiece.

Description

Technical Field

[0001] This disclosure relates to a control device, a mechanical system, and a control program. Background Technology

[0002] In recent years, robotic systems that use vision sensors to photograph objects (workpieces) transported by conveyors or other conveying devices, process the photographed images to detect the workpieces, and use the detected workpiece positions to enable robots to pick up (sort) the workpieces have been put into practical use.

[0003] In such a robotic system, when the robot fails to remove a workpiece from the conveyor, the image of the failed removal (the failure image) is used to adjust the image processing parameters. That is, by reviewing the history of camera images to determine if a workpiece was not detected, the image processing parameters are adjusted using images from cases where no workpiece was detected, thereby improving the workpiece detection accuracy.

[0004] In the past, various schemes have been proposed for studying the storage and processing of captured images in robot control devices (control devices) that capture images of workpieces transported by a conveying device and use those images to enable robots to perform prescribed processing on the workpieces. Existing technical documents Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 09-131683 Patent Document 2: Japanese Patent Application Publication No. 2020-032520 Patent Document 3: Japanese Patent Application Publication No. 2012-166308 Summary of the Invention The problem the invention aims to solve

[0006] As mentioned above, robotic systems that use vision sensors to photograph workpieces transported by conveyors and other means, and then enable robots to retrieve these workpieces, have been put into practical use. However, there are a large number of failed images where the robot was unable to retrieve a workpiece from the conveyor, making the process of adjusting image processing parameters using these numerous failed images extremely cumbersome.

[0007] Furthermore, the parameter adjustment of image processing is not limited to the robot system that performs the above-mentioned workpiece removal. For example, it is also applicable to various robot systems (mechanical systems) that enable robots (machines) to perform specified processing and machining of workpieces.

[0008] Therefore, there is a need to provide a control device, mechanical system, and control program capable of efficiently adjusting parameters for image processing. Solution for solving the problem

[0009] According to one embodiment of this disclosure, a control device is provided to control machinery to perform a prescribed operation on a plurality of workpieces transported by a conveying device. The control device includes a queue management unit that manages the position of each workpiece based on the output of an image processing device and the output of a position detector. The image processing device processes images of the plurality of workpieces captured by a vision sensor and compares the image processing score for each workpiece with a threshold to determine the detection of that workpiece. The position detector detects the position of each workpiece transported by the conveying device.

[0010] The queue management unit acquires information about a first workpiece whose image processing score is above a first threshold, and information about a second workpiece whose image processing score is below the first threshold but above a second threshold, as well as information about an image of the second workpiece obtained by taking a picture of the second workpiece. Based on the output of the position detector and the information of the second workpiece, it determines the image to be used in the improvement of image processing. Attached Figure Description

[0011] Figure 1 This is a front view that schematically illustrates the overall structure of one embodiment of the robot system involved in this implementation. Figure 2 yes Figure 1 The diagram shows a top view of the robot system. Figure 3 It is shown Figure 1 and Figure 2 A functional block diagram of an example of a robot control device (control unit) in a robot system shown. Figure 4 This is a flowchart illustrating an example of the processing in one embodiment of the control program involved in this implementation. Figure 5 This is a flowchart illustrating another example of the processing in one embodiment of the control program involved in this implementation. Detailed Implementation

[0012] The following detailed description of embodiments of the control device, mechanical system, and control program involved in this embodiment is based on the accompanying drawings. In the drawings, the same or similar constituent elements are labeled with the same or similar reference numerals. Furthermore, the embodiments described below do not limit the technical scope of the invention as described in the claims or the meaning of the terms used. In addition, in this specification, the term "mechanical" includes various types of machinery, including industrial robots and collaborative robots. Furthermore, in this specification, the processing performed by the robot is not limited to the removal of workpieces moving on a conveyor, but also includes various processing and handling of moving workpieces.

[0013] Figure 1 This is a front view that schematically illustrates the overall structure of one embodiment of the robot system involved in this implementation. Figure 2 yes Figure 1 A top view of the robot system shown. Figure 1 and Figure 2 In the attached diagram, reference numeral 100 indicates a robot system (mechanical system), 1 indicates a robot (mechanical), 2 indicates a control device (robot control device), 3 indicates a camera device, 4 indicates an image processing device, 5 indicates a conveyor position detection device, 6 indicates a conveyor, and W indicates a workpiece. Furthermore, in... Figure 1 and Figure 2 In this system, the image processing device 4 is built into the robot control device 2, but it can also be set up separately and independently from the robot control device 2.

[0014] like Figure 1 and Figure 2 As shown, the robot system 100 includes a robot 1 equipped with a hand 10, a robot control device 2 with a built-in image processing device 4, a camera device 3, and a conveyor 6. The robot 1 includes: a base 14 fixed to a mounting surface; a rotating base 13 that rotates relative to the base 14; a lower arm 12 rotatably supported on the rotating base 13; an upper arm 11 rotatably supported on the lower arm 12; and a wrist 15 rotatably supported on the end of the upper arm 11. Here, the upper arm 11 rotates about a rotation axis parallel to the extending direction of the upper arm 11, and a rotatably formed flange 16 is provided at the front end of the wrist 15. Furthermore, in this embodiment, the robot 1 is described as a multi-joint robot with multiple joints, but it is not limited to this configuration; various working tools can be installed and any robot can be used. In addition, as an example, the workpiece W is set as a rectangular cardboard box, but the workpiece W is of course not limited to a cardboard box, and can also be of various materials and shapes.

[0015] The hand 10 is a working tool capable of holding the workpiece W, for example, by adsorbing and holding the workpiece W through multiple adsorption pads 2a. The hand 10 is fixed to the flange 16 of the wrist 15, but as a working tool mounted on the robot 1, any working tool can be used.

[0016] Conveyor 6 is an example of a conveying device for transporting workpiece W. It drives the annular belt 6a to rotate via a drive motor M, thereby transporting workpiece W in a predetermined direction. That is, as indicated by arrow TD, workpiece W is moved horizontally and transported to a position where robot 1 can change its position and posture so that its hand 10 can grasp workpiece W. Here, an encoder is installed in the drive motor M, which functions as a conveyor position detector 5 to detect the position of workpiece W. Furthermore, the conveyor position detector 5 is limited to the encoder installed in the drive motor M; for example, the movement of belt 6a can also be calculated directly.

[0017] The imaging device 3 includes, for example, a vision sensor (camera) 30 that captures at least one of a two-dimensional image and a three-dimensional image of the workpiece W. The image captured by the vision sensor 30 is output to the image processing device 4 (robot control device 2). Here, the imaging device 3 may be a two-dimensional camera with a field of view (FV), but various known vision sensors, such as a three-dimensional camera with two cameras and a light-emitting part of an image pattern, can be used.

[0018] The vision sensor 30 is supported by the support member 31 and is positioned to capture images of the workpiece W being transported by the conveyor 6. Furthermore, the vision sensor 30 is positioned upstream of the robot 1 in the direction TD in which the workpiece W is transported. Here, the robot system 100 is provided with a world coordinate system WC as a reference coordinate system; for example, the origin of the world coordinate system WC is set at the base 14 of the robot 1. Therefore, even if the position and posture of the robot 1 change, the position and orientation of the world coordinate system WC will not change. Furthermore, the world coordinate system WC has mutually orthogonal X-axis, Y-axis, and Z-axis as coordinate axes, and W-axis, P-axis, and R-axis as coordinate axes about the X-axis, Y-axis, and Z-axis.

[0019] Furthermore, in the robot system 100, a tool coordinate system TC is provided, with its origin set at any position of the working tool. The origin of the tool coordinate system TC is set at the tool tip point of the hand 10. Therefore, when the robot 1 changes its position and posture, the position and posture of the tool coordinate system TC change. For example, the position of the robot 1 corresponds to the position of the tool tip point, and the posture of the robot 1 corresponds to the orientation of the tool coordinate system TC relative to the world coordinate system WC. Additionally, in the robot system 100, a sensor coordinate system SC is provided corresponding to the vision sensor 30, with the origin of the sensor coordinate system SC fixed to the coordinate system of the vision sensor 30. Therefore, based on the position and posture of the sensor coordinate system SC relative to the world coordinate system WC, the coordinate values ​​in the sensor coordinate system SC can be transformed into coordinate values ​​in the world coordinate system WC.

[0020] Figure 3 It is shown Figure 1 and Figure 2 This is a functional block diagram of an example of a robot control device (control unit) in a robot system. Here, in... Figure 1 and Figure 2 In the robot system 100 shown, the robot control device 2 has a built-in image processing device 4, but... Figure 3 In the illustration, the robot control device 2 and the image processing device 4 are depicted separately. That is, the image processing device 4 can be integrated into the robot control device 2, or it can be separately disposed outside the robot control device 2.

[0021] exist Figure 3 In the attached diagram, 1 represents the robot, 2 represents the robot control device, 4 represents the image processing device, 5 represents the conveyor position detector, and 30 represents the vision sensor. Figure 3 As shown, the robot control device 2 controls, for example, the robot 1 to perform prescribed operations such as retrieving workpieces W transported by the conveyor 6, and includes an action program 21, an image processing trigger unit 22, a queue management unit 23, a queue 24, an action control unit 25, and a robot drive unit 26. Furthermore, the image processing device 4 processes multiple images of workpieces W captured by the vision sensor 30 and compares the image processing score for each workpiece W with a threshold to determine the detection of that workpiece W, and includes a detection unit 41, a result management unit 42, an image buffer 43, and an image storage unit 44. Here, the image buffer 43 and the image storage unit 44 constitute a storage device for storing (holding) image data, etc.

[0022] The motion program 21 controls the robot 1 to perform prescribed operations on the workpieces W transported by the conveyor 6. This program is stored in a storage unit such as ROM (Read Only Memory) or flash memory and executed by a microprocessor (MPU) that functions as a processing unit. Furthermore, the processing unit also performs functions such as the queue management unit 23 and the motion control unit 25. The image processing trigger unit 22 receives the output of the conveyor position detector 5, which detects the position of each workpiece W transported by the conveyor 6, and triggers the image processing device 4. The image processing device 4 then triggers the vision sensor 30.

[0023] The queue management unit 23 receives the outputs from the image processing device 4 (result management unit 42) and the conveyor position detector 5, and manages the position of each workpiece W. Specifically, the queue management unit 23 receives from the image processing device 4 information about successfully detected workpieces (first workpieces) W whose image processing score is above a first threshold, and also receives from the image processing device 4 information about specific failed-detection workpieces (second workpieces) W whose image processing score is below the first threshold but above a second threshold, as well as information about specific failed-detection images (second workpiece images) obtained by capturing images of the specific failed-detection workpieces W.

[0024] Furthermore, the queue management unit 23 determines the image used for image processing improvement in the image processing device 4 based on the output of the conveyor position detector 5 and information about the specific failed detection workpiece W. The queue 24 receives the output of the queue management unit 23 and stores information about the successfully detected workpiece W, information about the specific failed detection workpiece W, and information for determining the image that was detected. The motion control unit 25 generates motion commands for controlling the robot 1 based on the motion program 21 and the output of the queue management unit 23. The robot drive unit 26 drives and controls the robot 1 based on the motion commands output from the motion control unit 25.

[0025] The detection unit 41 receives multiple images of workpieces W captured by the vision sensor 30, compares the image processing score for each workpiece W with a threshold for detection (determination), and outputs the determination result to the result management unit 42. Here, regarding the threshold used by the detection unit 41 to determine the image processing score for the workpiece W (e.g., template matching similarity: %), two thresholds are prepared: a first threshold (e.g., 90%) and a second threshold (e.g., 80%). The result management unit 42 receives the detection result obtained by the detection unit 41 by comparing it with the first and second thresholds, and outputs information about successfully detected workpieces W, information about specific failed-detection workpieces W, and information about specific failed-detection images to the queue management unit 23.

[0026] Specifically, if the image processing score for workpiece W is above a first threshold (90%), the image data of workpiece W is determined to be successfully detected image data; conversely, if the image processing score for workpiece W is below the first threshold (90%), the image data of workpiece W is determined to be undetected image data. Here, in the robot control device (image processing device) according to this embodiment, the image processing score for workpiece W is compared with a second threshold (80%) that is lower than the first threshold (90%) for detection.

[0027] That is, if the image processing score for workpiece W is less than a first threshold (90%) but greater than a second threshold (80%), the image data of workpiece W is determined as specific detection failure image data. Furthermore, if the image processing score for workpiece W is less than the second threshold (80%), the image data of workpiece W is directly determined as detection failure image data. Here, the specific detection failure image data of workpiece W with an image processing score less than the first threshold (90%) but greater than the second threshold (80%) is used, for example, for parameter adjustment of the image processing device 4, thereby improving the detection accuracy of workpiece W.

[0028] Setting the first threshold to 90% and the second threshold to 80% is just an example. The values ​​of the first and second thresholds can vary depending on the operation and processing content of the workpiece W by the robot system 100. That is, when using detection failure image data (specific detection failure image data) that is less than the first threshold and greater than the second threshold, the first and second thresholds are set to values ​​that allow for efficient parameter adjustment of the image processing device 4 in a short time.

[0029] Here, specific failed detection image data that is less than the first threshold but greater than the second threshold is input to a development computer located in an office separate from the factory where the robot 1 and robot control device 2 are installed, and the parameters of the image processing device 4 are adjusted to improve the detection accuracy of the workpiece W. Furthermore, image data from the image storage unit 44 and the queue 24, as well as the size and position information of the workpiece W, can be input to the development computer via a storage unit such as a USB memory, or a wired or wireless communication unit.

[0030] Firstly, in the robot control device 2 (image processing device 4) of the first embodiment of this implementation, the storage device consists of an image buffer 43 and an image storage unit 44. Based on the output of the result management unit 42, it stores at least the specific failure image data obtained by photographing specific failure workpieces W. That is, in the first embodiment of this implementation, the image buffer 43 holds the image data obtained by photographing each workpiece W, and the image storage unit 44 only stores the specific failure image data obtained by photographing specific failure workpieces W whose image processing score is less than a first threshold and greater than a second threshold.

[0031] Furthermore, in the robot control device 2 of the first embodiment, the queue management unit 23 writes the position of each workpiece W along with the value of the conveyor position detector 5 when the image of the workpiece W is acquired to the queue 24. When saving to the queue 24, the queue management unit 23 determines whether the value of the conveyor position detector 5 is within the allowable range if it is the same as the value of the conveyor position detector 5 of the successfully detected workpiece (first workpiece) W in the queue 24. If it is determined to be within the allowable range, it is determined to be the same workpiece W and one of the results is discarded. Additionally, in the case of a specific failed detection workpiece (second workpiece) W, the queue management unit 23 stores the specific failed detection image data (second workpiece image data) in the image buffer 43 and writes it to the queue 24 as a failed detection workpiece W.

[0032] The queue management unit 23 determines whether there is a workpiece W that is the same as the newly saved successfully detected workpiece W among the failed workpieces W in the queue 24. If it determines that there is, it discards the failed workpiece W in the queue 24. Furthermore, when all the failed workpieces W in the queue 24 are discarded, it discards the image data held in the image buffer 43. Then, based on the current position calculated according to the size information of each workpiece W and the value of the conveyor position detector 5, the queue management unit 23 saves the image data held in the image buffer 43 when the workpiece W is completely out of the field of view (outside the camera range of the vision sensor 30) as specific failed detection image data in the image storage unit 44, and determines this image data as the image used for image processing improvement in the image processing device 4.

[0033] Here, the image processing device 4 performs template matching, for example, comparing the shapes of each workpiece W captured by the vision sensor 30 with pre-prepared shape data of the workpiece W, calculating the similarity (score) between the two, and comparing it with thresholds (a first threshold and a second threshold). However, the comparison between the image processing score and the threshold is not limited to the score (similarity) based on template matching; scores from various image processing methods can be compared with the thresholds. In this way, according to the robot control device 2 (image processing device 4) of this first embodiment, the image storage unit 44 only needs to store, for example, the image data used in the parameter adjustment of the image processing device 4 (specific detection failure image data), thus significantly reducing the memory capacity of the storage device (image storage unit 44). Furthermore, the parameter adjustment of the image processing device can be performed, for example, based on the output of the conveyor position detector 5, information on the specific detection failure workpiece W, and the image determined by the robot control device 2.

[0034] Next, in the robot control device 2 (image processing device 4) according to the second embodiment of this invention, the storage device is composed of an image storage unit 44 that stores all image data obtained by photographing each workpiece W. That is, in the second embodiment of this invention, the image storage unit 44 stores all image data including specific failure image data obtained by photographing specific failure workpieces W whose image processing scores are less than a first threshold and more than a second threshold. Furthermore, in this second embodiment, it is not necessary to provide the image buffer 43 described in the first embodiment as a storage device.

[0035] In this second embodiment, the queue management unit 23 writes the position of each workpiece W along with the value of the conveyor position detector 5 when the image of the workpiece W is acquired to the queue 24, and saves all image data obtained by photographing each workpiece W in the image storage unit 44. Here, the image data obtained by photographing each workpiece W and saved to the image storage unit 44 is determined based on the value of the conveyor position detector 5 in a way that makes the range captured by the vision sensor 30 continuous.

[0036] That is, the image processing trigger unit 22 identifies that the conveyor 6 has moved a certain distance (the camera range of the vision sensor 30) based on the output of the conveyor position detector 5, outputs a trigger signal, and saves all image data in the image storage unit 44 in a continuous manner, according to the images captured by each camera range (field of view) of the vision sensor 30. As a result, the image storage unit 44 not only stores specific detection failure image data, but also detection success image data and all detection failure image data, thus requiring a large memory capacity. Here, all image data stored in the image storage unit 44 is input to a development computer set up in an office or the like, separated from the factory, for example via a USB memory or communication unit, where data used for parameter adjustment of the image processing device 4 is selected.

[0037] Furthermore, in the user interface that displays historical images sequentially from oldest to newest, when a specific failed detection workpiece W exists, the queue management unit 23 determines an image containing that workpiece W based on the current position calculated according to the size information of each workpiece W and the value of the conveyor position detector 5, and also compares and detects the determined images. Then, when no successfully detected image data for workpiece W, which is determined to be the same workpiece based on the value of the conveyor position detector 5 corresponding to the image data stored in the image storage unit 44, is found, the queue management unit 23 determines these image data as images to be used for image processing improvement in the image processing device 4.

[0038] As described above, in the first and second embodiments of the control device (robot control device) involved in this embodiment, the image processing score can be determined based on the similarity of the template matching of the workpiece in the captured image, but is not limited to the similarity of the template matching of the workpiece, and can apply the scores of various known image processing methods.

[0039] Figure 4 This is a flowchart illustrating an example of the processing in one embodiment of the control program involved in this embodiment, used to explain the control processing during workpiece inspection in the first embodiment of this embodiment. For example... Figure 4 As shown, when the control process for workpiece detection in one embodiment of the control program involved in this embodiment starts (START), in step ST11, the image processing trigger unit 22 applies a trigger to the vision sensor 30 to capture an image (the image processing trigger unit 22 applies a trigger to the image processing device 4, and the image processing device 4 applies an image trigger to the vision sensor 30), and proceeds to step ST12. Furthermore, as described above, the detection unit 41 of the image processing device 4 receives images of multiple workpieces W captured by the vision sensor 30, and compares the image processing score for each workpiece W with a threshold for detection.

[0040] That is, in step ST12, the detection unit 41 acquires an image from the vision sensor 30, compares the image processing score for each workpiece W with the first threshold and the second threshold for detection (determination), and outputs the determination result to the result management unit 42. Here, as described above, in the first embodiment of this implementation, workpieces that fail detection (specific workpieces that fail detection) are detected if they are less than the first threshold and greater than the second threshold, and proceed to step ST13.

[0041] In step ST13, the result management unit 42 of the image processing device 4 attaches an ID (identification number) to the image (image data) and saves it to the image buffer 43. The result management unit 42 of the image processing device 4 also sends the information on successfully detected workpieces, the information on failed workpieces, and the image ID to the queue management unit 23 of the robot control device 2. Here, as described above, the information on successfully detected workpieces refers to those whose image processing score for workpiece W is above a first threshold. The information on failed workpieces refers to those whose image processing score for workpiece W is below the first threshold. Furthermore, specific failed workpieces whose image processing score is below the first threshold but above a second threshold are naturally included among those whose image processing score is below the first threshold.

[0042] Next, proceeding to step ST14, the queue management unit 23 saves only the successfully detected workpieces that are not considered the same workpiece in queue 24, saves the image ID in queue 24, saves the failed workpieces in queue 24 in a way that associates them with the image ID, deletes the failed workpieces that are considered the same workpiece in queue 24 from queue 24, and then notifies the result management unit 42 of the image processing device 4 of the IDs of all the failed workpieces that have been associated with the deleted workpieces.

[0043] Furthermore, proceeding to step ST15, the result management unit 42 of the image processing apparatus 4 deletes the image with the received notification ID from the image buffer 43, thus ending the control processing during workpiece inspection in one embodiment of the control program involved in the implementation. Here, refer to Figure 4 In one embodiment of the control program described herein, the image data stored in the image storage unit 44 is set only to image data of a specific failed detection workpiece that is preferred for efficient image processing parameter adjustment, i.e., image data of a specific failed detection workpiece whose image processing score is less than a first threshold and more than a second threshold. This can significantly reduce the memory capacity of the image storage unit 44.

[0044] Figure 5 This is a flowchart illustrating another example of the processing in one embodiment of the control program involved in this implementation, used to explain the control processing for determining the current position of the workpiece. For example... Figure 5 As shown, when the control process for determining the current position of the workpiece in one embodiment of the control program involved in this embodiment starts (START), in step ST21, the queue management unit 23 determines whether the workpiece W as a whole has left the field of view of the vision sensor 30 based on the current position and shape information of the failed detection workpieces in the queue 24. That is, when it is determined in step ST22 that the workpiece W as a whole has not left the field of view of the vision sensor 30 ("No"), the control process for determining the current position of the workpiece is directly ended (END).

[0045] On the other hand, when it is determined in step ST22 that the workpiece W has completely left the field of view of the vision sensor 30 ("Yes"), the process proceeds to step ST23, where the image ID associated with the corresponding failed workpiece W is deleted from queue 24, and the deleted image ID is notified to the result management unit 42 of the image processing device 4. Here, the failed workpiece W is the workpiece W whose image processing score is less than a first threshold. When the workpiece W has completely left the field of view of the vision sensor 30 (outside the camera range), the image ID associated with the failed workpiece W is deleted from queue 24, and the image data corresponding to the deleted ID is notified to the result management unit 42.

[0046] Furthermore, proceeding to step ST24, the result management unit 42 of the image processing device 4 saves the image (image data) corresponding to the ID to the image storage unit 44, and the result management unit 42 of the image processing device 4 deletes the image from the image buffer 43, thus ending the control processing for determining the current position of the workpiece. Additionally, refer to... Figure 4 and Figure 5 The processing of one embodiment of the control program described in this embodiment is merely an example, and various changes and modifications are of course possible.

[0047] The control program described in this embodiment can be provided by recording on a computer-readable non-transitory recording medium or a non-volatile semiconductor memory, or it can be provided via wired or wireless means. Examples of computer-readable non-transitory recording media include CD-ROMs (Compact Disc Read Only Memory), DVD-ROMs, and hard disk drives. Examples of non-volatile semiconductor memories include PROMs (Programmable Read Only Memory) and flash memory. Furthermore, as a distribution from a server device, it can be provided via a wired or wireless LAN (Local Area Network) or WAN (Wide Area Network) such as the Internet.

[0048] As described in detail above, the control device, mechanical system, and control program involved in this embodiment enable efficient adjustment of image processing parameters.

[0049] This disclosure has been described in detail, but it is not limited to the various embodiments described above. Various additions, substitutions, modifications, and partial deletions can be made to these embodiments without departing from the spirit of this disclosure or from the spirit of this disclosure derived from the claims and their equivalents. Furthermore, these embodiments can also be implemented in combination. For example, in the embodiments described above, the order of each action and each process is shown as an example and is not limited to these orders. The same applies to cases where numerical values ​​or formulas are used in the description of the embodiments described above.

[0050] The following notes further disclose the above-described embodiments and variations. [Postscript 1] A control device (2) controls a mechanism (1) to perform a prescribed operation on a plurality of workpieces (W) conveyed by a conveying device (6), wherein, The device includes a queue management unit (23) that manages the position of each workpiece (W) based on the output of an image processing device (4) and a position detector (5). The image processing device (4) processes images of multiple workpieces (W) captured by a vision sensor (30) and compares the image processing score for each workpiece (W) with a threshold to determine the detection of that workpiece (W). The position detector (5) detects the position of each workpiece (W) being transported by the conveying device (6). The queue management unit (23) acquires information about a first workpiece (W) whose image processing score is above a first threshold, and acquires information about a second workpiece (W) whose image processing score is below the first threshold but above a second threshold, as well as information about the second workpiece image obtained by capturing the second workpiece (W). The queue management unit (23) determines the image to be used in the image processing improvement based on the output of the position detector (5) and the information of the second workpiece (W). [Postscript 2] According to the control device described in Appendix 1, wherein, It also includes an image processing trigger unit (22), which receives the output of the position detector (5) and applies a trigger to the image processing device (4), and the image processing device (4) applies an image trigger to the vision sensor (30). [Postscript 3] According to the control device described in Appendix 1 or 2, wherein, The control device (2) has the image processing device (4) built in. [Postscript 4] The control device according to any one of Appendices 1 to 3, wherein... The image processing device (4) has: The detection unit (41) receives images of a plurality of workpieces (W) captured by the vision sensor (30) and compares the image processing score for each workpiece (W) with the first threshold and the second threshold for detection. The result management unit (42) receives the detection result obtained by the detection unit (41) by comparing it with the first threshold and the second threshold, and outputs information about the first workpiece (W), information about the second workpiece (W), and information about the image of the second workpiece to the queue management unit (23) of the control device (4); and The storage devices (43, 44) store at least the image data of the second workpiece obtained by photographing the second workpiece (W) based on the output of the result management unit (42). [Postscript 5] According to the control device described in Appendix 4, wherein... The storage device includes: An image buffer (43) holds image data obtained by capturing images of each of the workpieces (W); and The image storage unit (44) only stores the image data of the second workpiece (W) obtained by taking pictures of the second workpiece (W) whose score for the image processing is less than the first threshold and greater than the second threshold. [Postscript 6] According to the control device described in Appendix 5, wherein... The queue management department (23) performs the following processing: The position of each workpiece (W) and the value of the position detector (5) when the image of the workpiece (W) is acquired are written together into the queue (24); When saving to the queue (24), if the value of the position detector (5) is the same as the value of the position detector (5) of the first workpiece (W) in the queue (24), it is determined whether it is within the allowable range. If it is determined to be within the allowable range, it is determined to be the same workpiece (W) and one of the results is discarded. In the presence of the second workpiece (W), the image data of the second workpiece is held in the image buffer (43) and written to the queue (24) as a workpiece (W) that has failed to be detected. Determine whether there is a workpiece (W) in the queue (24) that is the same as the newly saved first workpiece (W). If it is determined that there is, discard the workpiece (W) in the queue (24). When all the failed workpieces (W) in the queue (24) are discarded, the image data held in the image buffer (43) is discarded; and Based on the current position calculated according to the size information of each workpiece (W) and the value of the position detector (5), the image data held in the image buffer (43) when the workpiece (W) is completely out of the field of view is saved as the second workpiece image data to the image storage unit (44), and the image data is determined as the image used in the improvement of image processing. [Postscript 7] According to the control device described in Appendix 5, wherein... The storage device includes an image storage unit (44) that stores all image data obtained by photographing each of the workpieces (W), including image data of the second workpiece obtained by photographing the second workpiece (W) whose image processing score is less than the first threshold and greater than the second threshold. [Postscript 8] According to the control device described in Appendix 7, wherein... The queue management unit (23) writes the position of each workpiece (W) and the value of the position detector (5) when the image of the workpiece (W) is acquired into the queue (24), and saves all image data obtained by taking pictures of each workpiece (W) in the image storage unit (44). In the user interface that displays historical images sequentially from old to new, the queue management unit (23) determines an image containing the workpiece (W) based on the current position calculated according to the size information of each workpiece (W) and the value of the position detector (5) when the second workpiece (W) is present, and performs the comparison detection on the determined image. If there is no image data of the workpiece (W) that is determined to be the same workpiece based on the value of the position detector (5) corresponding to the image data stored in the image storage unit (44), the image data is determined as the image used for image processing improvement in the image processing device (4). [Postscript 9] According to the control device described in Appendix 8, wherein... All image data obtained by photographing each of the workpieces (W) and stored in the image storage unit (44) are determined based on the value of the position detector (5) in a manner that makes the range captured by the vision sensor (30) continuous. [Postscript 10] The control device according to any one of Appendices 1 to 9, wherein... The machine in question is a robot (1). The conveying device (6) is a conveyor. The control device is a robot control device (2) that controls the robot (1) based on an action program (21) to enable the robot (1) to perform prescribed operations on each of the workpieces (W) transported by the conveyor. The position detector is a conveyor position detector (5) that detects the position of the workpiece (W) moving on the conveyor. [Postscript 11] The control device according to Appendix 10 further comprises: The queue (24) stores information about the first workpiece (W), information about the second workpiece (W), and information for determining the image that has been detected, based on the output of the queue management unit (23). The motion control unit (25) generates motion commands for controlling the actions of the robot (1) based on the output of the motion program (21) and the queue management unit (23); and The robot drive unit (26) drives the robot (1) based on the motion commands output from the motion control unit. [Postscript 12] The control device according to any one of Appendices 1 to 11, wherein, The image processing score is determined based on the similarity of template matching of the workpiece (W) in the image captured by the vision sensor (30). [Postscript 13] The control device according to any one of Appendices 1 to 12, wherein, The parameters in the image processing device (4) are adjusted based on the output of the position detector (5), the information of the second workpiece (W), and the image determined by the control device (2). [Postscript 14] A mechanical system (100) includes: a machine (1); a control device (2) that controls the machine (1) to perform a prescribed operation on a plurality of workpieces (W) conveyed by a conveying device (6); and an image processing device (4) that processes images of the plurality of workpieces (W) captured by a vision sensor (30) and compares the image processing score for each workpiece (W) with a threshold to determine the detection of the workpiece (W), wherein, The control device (2) includes a queue management unit (23), which manages the position of each workpiece (W) based on the output of the image processing device (4) and the output of the position detector (5), wherein the position detector (5) detects the position of each workpiece (W) being transported by the conveying device (6). The queue management unit (23) acquires information about a first workpiece (W) whose image processing score is above a first threshold, and acquires information about a second workpiece (W) whose image processing score is below the first threshold but above a second threshold, as well as information about the second workpiece image obtained by capturing the second workpiece (W). The queue management unit (23) determines the image to be used in the image processing improvement based on the output of the position detector (5) and the information of the second workpiece (W). [Postscript 15] A control program in a mechanical system (100) determines an image used in image processing improvement, the mechanical system (100) comprising: a machine (1); a control device (2) that controls the machine (1) to perform a prescribed operation on a plurality of workpieces (W) conveyed by a conveying device (6); and an image processing device (4) that processes images of the plurality of workpieces (W) captured by a vision sensor (30) and compares the image processing score for each workpiece (W) with a threshold to determine the detection of the workpiece (W), wherein the control program causes the computational processing device to perform the following steps: The positions of each workpiece (W) are managed based on the output of the image processing device (4) and the output of the position detector (5), wherein the position detector (5) detects the position of each workpiece (W) being transported by the conveying device (6); and Information is obtained about a first workpiece (W) whose image processing score is above a first threshold, and information is obtained about a second workpiece (W) whose image processing score is below the first threshold but above a second threshold, as well as information about an image of the second workpiece (W) obtained by taking a picture of the second workpiece (W). Explanation of reference numerals in the attached figures

[0051] 1: Robot (Mechanical) 2: Robot control device (control unit) 3: Camera device 4: Image processing device 5: Conveyor position detector (position detector) 6: Conveyor (transfer device) 21: Action Procedure 22: Image Processing Trigger Unit 23: Queue Management Department 24: Queue 25: Motion Control Department 26: Robot Drive Department 30: Visual Sensor 41: Testing Department 42: Results Management Department 43: Image Buffer 44: Image Storage Department 100: Robotic System (Mechanical System) FV: Camera range (field of view) SC: Sensor coordinate system TC: Tool Coordinate System W: Workpiece (object) WC: World Coordinate System

Claims

1. A control device that controls machinery to perform predetermined operations on a plurality of workpieces conveyed by a conveying device, wherein, The system includes a queue management unit that manages the position of each workpiece based on the output of an image processing device and the output of a position detector. The image processing device processes images of multiple workpieces captured by a vision sensor and compares the image processing score for each workpiece with a threshold to determine the detection of that workpiece. The position detector detects the position of each workpiece being transported by the conveying device. The queue management unit acquires information about a first workpiece whose image processing score is above a first threshold, and acquires information about a second workpiece whose image processing score is below the first threshold but above a second threshold, as well as information about the second workpiece image obtained by capturing the second workpiece. The queue management unit determines the image to be used in the image processing improvement based on the output of the position detector and the information of the second workpiece.

2. The control device according to claim 1, wherein, It also includes an image processing trigger unit, which receives the output of the position detector and applies a trigger to the image processing device, and the image processing device applies an image trigger to the vision sensor.

3. The control device according to claim 1 or 2, wherein, The control device has the image processing device built into it.

4. The control device according to any one of claims 1 to 3, wherein, The image processing device has: The detection unit receives images of multiple workpieces captured by the vision sensor and compares the image processing score for each workpiece with the first threshold and the second threshold for detection. The result management unit receives the detection result obtained by the detection unit by comparing it with the first threshold and the second threshold, and outputs the information of the first workpiece, the information of the second workpiece, and the information of the second workpiece image to the queue management unit of the control device. as well as A storage device, based on the output of the result management unit, stores at least the image data of the second workpiece obtained by photographing the second workpiece.

5. The control device according to claim 4, wherein, The storage device includes: An image buffer that holds image data obtained from capturing images of each of the workpieces; and The image storage unit only stores the image data of the second workpiece obtained by taking pictures of the second workpiece whose score for the image processing is less than the first threshold and greater than the second threshold.

6. The control device according to claim 5, wherein, The queue management department performs the following processing: The position of each workpiece, along with the value of the position detector when the image of the workpiece is acquired, is written into a queue; When saving to the queue, if the value of the position detector is the same as the value of the position detector of the first workpiece in the queue, it is determined whether it is within the allowable range. If it is determined to be within the allowable range, it is determined to be the same workpiece and one of the results is discarded. In the presence of the second workpiece, the image data of the second workpiece is kept in the image buffer and written to the queue as a workpiece that failed to be detected; Determine whether there is a workpiece in the queue that is the same as the newly saved first workpiece among the failed workpieces. If it is determined that there is, discard the failed workpiece in the queue. When all the failed workpieces in the queue are discarded, the image data held in the image buffer is discarded. as well as Based on the current position calculated according to the size information of each workpiece and the value of the position detector, the image data held in the image buffer when the workpiece is completely out of the field of view is saved as the second workpiece image data to the image storage unit, and the image data is determined as the image used for image processing improvement.

7. The control device according to claim 5, wherein, The storage device includes an image storage unit that stores all image data obtained by photographing each of the workpieces, including image data of the second workpiece obtained by photographing the second workpiece whose image processing score is less than the first threshold and greater than the second threshold.

8. The control device according to claim 7, wherein, The queue management unit writes the position of each workpiece and the value of the position detector when the image of the workpiece is acquired into the queue, and saves all image data obtained by capturing images of each workpiece in the image storage unit. In a user interface that displays historical images sequentially from old to new, the queue management unit determines an image containing the second workpiece based on the current position calculated according to the size information of each workpiece and the value of the position detector when the second workpiece is present. The unit also performs the comparison detection on the determined image. If no image data of the workpiece that is determined to be the same workpiece based on the value of the position detector corresponding to the image data stored in the image storage unit exists, the unit determines these image data as images used for improving image processing in the image processing device.

9. The control device according to claim 8, wherein, All image data obtained by photographing each of the workpieces and stored in the image storage unit are determined based on the value of the position detector in a manner that makes the range captured by the vision sensor continuous.

10. The control device according to any one of claims 1 to 9, wherein, The machine in question is a robot. The conveying device is a conveyor. The control device is a robot control device that controls the robot based on motion programs to enable the robot to perform prescribed operations on each of the workpieces transported by the conveyor. The position detector is a conveyor position detector that detects the position of the workpiece moving on the conveyor.

11. The control device according to claim 10, further comprising: The queue, based on the output of the queue management unit, stores information about the first workpiece, information about the second workpiece, and information for determining the image that has been detected; The motion control unit generates motion commands for controlling the robot's actions based on the motion program and the output of the queue management unit. as well as A robot drive unit that drives the robot based on motion commands output from the motion control unit.

12. The control device according to any one of claims 1 to 11, wherein, The image processing score is determined based on the similarity of template matching of the workpiece in the image captured by the vision sensor.

13. The control device according to any one of claims 1 to 12, wherein, The parameters in the image processing device are adjusted based on the output of the position detector, the information of the second workpiece, and the image determined by the control device.

14. A mechanical system comprising: a machine; a control device that controls the machine to perform a predetermined operation on a plurality of workpieces conveyed by a conveying device; and an image processing device that processes images of the plurality of workpieces captured by a vision sensor and compares the image processing score for each of the workpieces with a threshold to determine the detection of the workpiece, wherein... The control device includes a queue management unit that manages the position of each workpiece based on the output of the image processing device and the output of the position detector, wherein the position detector detects the position of each workpiece being transported by the conveying device. The queue management unit acquires information about a first workpiece whose image processing score is above a first threshold, and acquires information about a second workpiece whose image processing score is below the first threshold but above a second threshold, as well as information about the second workpiece image obtained by capturing the second workpiece. The queue management unit determines the image to be used in the image processing improvement based on the output of the position detector and the information of the second workpiece.

15. A control program in a mechanical system for determining an image to be used in improving image processing, the mechanical system comprising: a machine; a control device that controls the machine to perform a prescribed operation on a plurality of workpieces conveyed by a conveying device; and an image processing device that processes images of the plurality of workpieces captured by a vision sensor and compares a score of the image processing for each of the workpieces with a threshold to determine the detection of the workpiece, wherein... The control program causes the computing device to perform the following steps: The positions of each workpiece are managed based on the output of the image processing device and the output of the position detector, wherein the position detector detects the position of each workpiece transported by the conveying device; and Information about a first workpiece whose image processing score is above a first threshold is obtained, and information about a second workpiece whose image processing score is below the first threshold but above a second threshold is obtained, as well as information about an image of the second workpiece obtained by taking a picture of the second workpiece.

Images