Gripping device and control method

Abstract

To improve a gripping success rate of a workpiece.SOLUTION: A control device 50 acquires each gripping success rate of at least one workpiece, on the basis of a first image photographed by a first camera 11. The control device 50 determines a target workpiece from at least the one workpiece on the basis of each gripping success rate. The control device 50 moves a gripping unit 30 until a distance between a hand 31 and the target workpiece becomes a first determined distance. The control device acquires a second image by moving the gripping unit 30, and then photographing by a second camera 12. The control device 50 acquires a first gripping position of the target workpiece, on the basis of the second image. The control device 50 outputs a control signal for gripping the target workpiece at the first gripping position by the gripping unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This invention relates to a gripping device and a control method.

Background Art

[0002] For example, Patent Document 1 discloses a gripping device for gripping a workpiece. This gripping device has an imaging device and a learned model constructed by machine learning. The gripping device acquires an image by the imaging device imaging the workpiece. The gripping device detects the optimal gripping position of the workpiece based on the image and the learned model. Then, the gripping device grips the workpiece at the optimal gripping position.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a gripping device, it is desired to improve the gripping success rate of a workpiece.

[0005] This invention has been made to solve the above problems, and it is to provide a technique for improving the gripping success rate of a workpiece.

Means for Solving the Problems

[0006] The gripping device of this disclosure comprises a first camera, a gripping unit, a second camera, and a control device. The first camera images at least one workpiece. The gripping unit has a hand for gripping the workpiece. The second camera is positioned on the gripping unit and images the hand. The control device controls the first camera, the second camera, and the gripping unit. Based on a first image captured by the first camera, the control device obtains the success rate of gripping each of the at least one workpiece. Based on the success rate of gripping each, the control device determines a target workpiece from the at least one workpiece. The control device moves the gripping unit until the distance between the hand and the target workpiece is a first predetermined distance. After moving the gripping unit, the control device obtains a second image by having the second camera capture an image. Based on the second image, the control device obtains a first gripping position of the target workpiece. The control device then outputs a control signal to cause the gripping unit to grip the target workpiece at the first gripping position.

[0007] Furthermore, the control method of this disclosure is a control method for a gripping device. The gripping device comprises a first camera, a gripping unit, and a second camera. The first camera images at least one workpiece. The gripping unit has a hand for gripping workpieces. The second camera is positioned on the gripping unit and images the hand. The control method includes obtaining the success rate of gripping each of at least one workpiece based on a first image captured by the first camera. The control method also includes determining a target workpiece from at least one workpiece based on the success rate of gripping each workpiece. The control method also includes moving the gripping unit until the distance between the hand and the target workpiece is a first predetermined distance. The control method also includes obtaining a second image by having the second camera capture an image after moving the gripping unit. The control method also includes obtaining a first gripping position of the target workpiece based on the second image. Finally, the control method includes outputting a control signal to cause the gripping unit to grip the target workpiece at the first gripping position. [Effects of the Invention]

[0008] According to the present invention, the success rate of gripping a workpiece can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of the gripping device. [Figure 2] This is a diagram showing a hand and other objects viewed from a planar perspective on the XZ plane. [Figure 3] This is a plan view of the hand and other components in the YZ plane. [Figure 4] This is a functional block diagram of the control unit. [Figure 5] This is a diagram showing an example of the first image. [Figure 6] This figure shows an example of the concept of the first position model. [Figure 7] This figure shows an example of the concept of a gripping success rate model. [Figure 8] This is a diagram showing an example of the second image. [Figure 9] This figure shows an example of the concept of the second position model. [Figure 10] This is a diagram showing an example of the third image. [Figure 11] This figure shows an example of a binarized second image. [Figure 12] This figure shows an example of a binarized second image. [Figure 13] This figure shows an example of updating the judgment data in the judgment data storage unit. [Figure 14] This is a flowchart showing the processing of the gripping device. [Figure 15] This is a plan view of a hand or other component from another embodiment, in the XZ plane. [Figure 16] This is a plan view of a hand or other component from another embodiment, in the YZ plane. [Figure 17] This figure shows a third image when a hand of a different embodiment is used. [Figure 18] This figure shows an example of a hand that employs a suction type mechanism. [Figure 19]This is a diagram showing a third image when a hand with a suction type is adopted. [Figure 20] This is a diagram showing another aspect of the hand 31 with a suction type adopted. [Figure 21] This is a diagram showing the lighting device as viewed in a plan view from the optical axis of the second camera.

Embodiments for Carrying out the Invention

[0010] Hereinafter, embodiments of the present invention will be described based on the drawings. In the following drawings, the same or corresponding parts are denoted by the same reference numerals and their descriptions will not be repeated.

[0011] [Configuration of Gripping Device] FIG. 1 is a diagram for explaining an overall configuration example of a gripping device 100. The gripping device 100 includes a first camera 11, a second camera 12, a gripping unit 30, and a control device 50. The gripping unit 30 has a positioning mechanism 32 and a hand 31 disposed at the tip of the positioning mechanism 32. In the example of FIG. 1, a workbench 60 is installed. At least one workpiece is disposed on the workbench 60. Thus, at least one workpiece is disposed in a predetermined area.

[0012] The first camera 11 is supported by a support portion 15. The first camera 11 is installed above the workbench 60. The first camera 11 captures an image of an imaging range including at least one workpiece disposed on the workbench 60 from above. The first camera 11 is, for example, a two-dimensional imaging device.

[0013] The hand 31 grips the workpiece W. The detailed configuration of the hand 31 will be described later. The positioning mechanism 32 adjusts the posture and position of the hand 31 so that the hand 31 can grip the workpiece on the workbench 60. The positioning mechanism 32 is, for example, a multi-axis robot.

[0014] The gripping unit 30 is in its initial position. When the gripping unit 30 is in its initial position, it is preferable that the gripping unit 30 is not included in the imaging range of the first camera 11. When the user performs a start operation to initiate the gripping process on the gripping device 100, the gripping unit 30 starts the gripping process, which will be described later.

[0015] The second camera 12 is located on the gripping unit 30. Typically, the second camera 12 is at the tip of the positioning mechanism 32 and mounted near the hand 31. The second camera 12 images, for example, the hand 31 and at least a portion of the workpiece when the hand 31 grips the workpiece. The second camera 12 is, for example, a two-dimensional imaging device.

[0016] The control device 50 controls the first camera 11, the second camera 12, and the gripping unit 30. The control device 50 also mainly includes a CPU 52 (Central Processing Unit) and memory 54. The memory 54 includes, for example, ROM (Read Only Memory) and RAM (Random Access Memory). The ROM stores the program executed by the CPU 52. The RAM temporarily stores data generated by the execution of the program by the CPU 52.

[0017] Figures 2 and 3 are diagrams illustrating examples of the configuration of the hand 31 and the like. The hand 31 in this embodiment is composed of a so-called chuck mechanism. The hand 31 has a gripping portion 31A, a gripping portion 31B, a fixing portion 31C, a fixing portion 31D, and an opening / closing portion 31E.

[0018] The gripping portion 31A is fixed to the fixing portion 31C. The gripping portion 31B is fixed to the fixing portion 31D. The fixing portions 31C and 31D are attached to the opening / closing portion 31E.

[0019] The opening / closing section 31E moves the fixing section 31C and fixing section 31D in a direction that brings them closer together or further apart in a predetermined direction. This predetermined direction corresponds to the "first predetermined direction" in this disclosure. This allows the gripping section 31A and gripping section 31B to be opened and closed. When the gripping unit 30 grips a workpiece, the control device 50 moves the gripping section 31A and gripping section 31B in a closing direction (a direction that brings the gripping section 31A and gripping section 31B closer together). When the gripping unit 30 releases the gripped workpiece, the control device 50 moves the gripping section 31A and gripping section 31B in an opening direction (a direction that moves the gripping section 31A and gripping section 31B further apart).

[0020] The second camera 12 is installed at the tip 32A of the positioning mechanism 32. The second camera 12 is positioned so that its optical axis is perpendicular to the first predetermined direction. Furthermore, the optical axis of the second camera 12 coincides with the central axis of the tip 32A. Typically, the optical axis of the second camera 12 is the axis of symmetry of the lens of the second camera 12.

[0021] In the example shown in Figure 2, the opening and closing direction (first predetermined direction) of the gripping parts 31A and 31B is the X-axis, and the optical axis of the second camera 12 is the Z-axis. The axis perpendicular to the X-axis and Z-axis is the Y-axis. Figure 2 is a plan view of the hand 31 etc. in the XZ plane. Figure 3 is a plan view of the hand 31 etc. in the YZ plane. As shown in Figure 3, the gripping parts 31B (and gripping parts 31A) have a roughly L-shape. The number of gripping parts may be three or more. The shape of the gripping parts may also be other shapes.

[0022] [Functional block diagram of the control unit] Figure 4 is a functional block diagram of the control device 50. The control device 50 includes a first acquisition unit 101, a second acquisition unit 102, a first detection unit 104, a judgment data storage unit 118, an output unit 110, a second detection unit 124, a determination unit 130, a judgment unit 120, and a command unit 132.

[0023] The first detection unit 104 includes a first position estimation unit 108 and a first position model 106. The output unit 110 includes a success rate estimation unit 116, a construction unit 112, and a gripping success rate model 114. The second detection unit 124 includes a second position estimation unit 126 and a second position model 128. The first position model 106 corresponds to the "first estimation model" of this disclosure. The gripping success rate model 114 corresponds to the "second estimation model" of this disclosure. The second position model 128 corresponds to the "third estimation model" of this disclosure.

[0024] The first camera 11 generates a first image by imaging at least one workpiece W placed on the workbench 60. The first acquisition unit 101 acquires the first image from the first camera 11. The first image is output to the first position estimation unit 108. Based on the first image and the first position model 106, the first position estimation unit 108 acquires the temporary gripping position of each of the at least one workpiece. The temporary gripping position corresponds to the "second gripping position" in this disclosure.

[0025] The provisional gripping position is the position at which gripping is presumed to be successful if the gripping unit 30 grips the workpiece at that provisional gripping position. In addition, at least one workpiece is a workpiece captured by the first image. The first position model 106 is constructed in advance by machine learning. The first position model 106 is constructed using a neural network. The neural network consists of an input layer, a hidden layer, an output layer, and links connecting each layer. When the first image is input to the input layer of the first position model 106, the output layer outputs the provisional gripping position for all workpieces (workpiece images) included in the first image.

[0026] Figure 5 is an example of a first image. Such a first image is output to the first detection unit 104. Figure 6 is a diagram illustrating an example of the concept of the first position model 106. In the example in Figure 6, the first image data G1, G2, ..., Gn are associated with profiles A1, A1, ..., An, respectively. Hereinafter, n is an integer of 2 or more. The first image data G1, G2, ..., Gn will be collectively referred to as the first image data G, and the profiles A1, A1, ..., An will be collectively referred to as the profile A.

[0027] The first image data G is image data of images previously captured by the first camera 11 or a camera equivalent to the first camera, of one or more workpieces. In the example in Figure 6, the first image data G is shown as an example where there are three workpieces (workpieces A to C). However, the first image data G may also be prepared for one or more workpieces.

[0028] Profile data is data that defines the temporary gripping position of each workpiece on a pixel-by-pixel basis. The temporary gripping position is the gripping position for the gripping unit 30 to properly grip the workpiece. For example, in the profile data of the example in Figure 6, for the first image data G1 showing workpieces A to C, the temporary gripping positions of workpiece A, workpiece B, and workpiece C are specified.

[0029] The first position model 106 is constructed, for example, by manual input from an inputter. The inputter is either the developer of the gripping device 100 or a user of the gripping device 100. For example, the inputter causes the first camera 11 or a camera equivalent to the first camera to capture images of one or more workpieces to generate first image data. Then, the inputter displays the images related to the first image data on a display device (not shown) and specifies the temporary gripping position of the displayed workpiece.

[0030] In this way, the temporary gripping position specified by the inputter is used as training data, and the temporary gripping position and the first image data are used as learning data to construct the first position model 106.

[0031] Furthermore, as will be described later, the first position model 106 is designed to be automatically updateable. Note that the first position model 106 only needs to be able to detect the provisional grasping position from the first image data and does not necessarily need to be constructed using a neural network. For example, it may be constructed using machine learning methods other than neural networks.

[0032] Let's return to the explanation in Figure 4. The success rate estimation unit 116 outputs the gripping success rate for each of the temporary gripping positions based on all the temporary gripping positions (each temporary gripping position) output by the first position estimation unit 108 and the gripping success rate model 114. The gripping success rate is the probability that the gripping unit 30 will successfully grip the workpiece if the control device 50 outputs a control signal to the gripping unit 30 to grip the workpiece at the temporary gripping position. The gripping success rate model 114 is constructed, for example, using a neural network. The gripping success rate model 114 outputs the gripping success rate from its output layer when the temporary gripping positions are input to its input layer.

[0033] Figure 7 shows an example of the concept of the grasping success rate model 114. In the grasping success rate model 114 in Figure 7, m judgment data B1, B2, ..., Bm are defined, where m is an integer greater than or equal to 2. Hereafter, the judgment data B1, B2, ..., Bm will be collectively referred to as judgment data B.

[0034] In the judgment data B, the temporary gripping position and the gripping success / failure result are associated. The gripping success / failure result indicates whether the gripping unit 30 successfully gripped the workpiece when the control device 50 output a control signal to grip at the temporary gripping position associated with the gripping success / failure result.

[0035] In the example shown in Figure 7, for instance, in judgment data B1, the provisional gripping position is associated with the gripping success / failure result, indicating success. Similarly, in judgment data B2, the provisional gripping position is associated with the gripping success / failure result, indicating failure. The gripping success rate model 114 may be constructed by manual input from the user.

[0036] In this way, the input grasp success / failure result is used as training data, and the grasp success / failure result and the provisional grasp position are used as learning data to construct the grasp success rate model 114.

[0037] Furthermore, as will be described later, the grasping success rate model 114 is designed to be automatically updateable. Note that the grasping success rate model 114 only needs to be able to detect the grasping success rate from the provisional grasping position and does not necessarily need to be constructed using a neural network. For example, it may be constructed using machine learning methods other than neural networks.

[0038] Let's return to the explanation in Figure 4. The determination unit 130 determines the target workpiece from at least one workpiece imaged by the first camera 11, based on the gripping success rate from the success rate estimation unit 116. The target workpiece is the workpiece to be gripped by the gripping unit 30. For example, the determination unit 130 identifies a gripping success rate that is higher than a predetermined threshold from among all the gripping success rates output by the success rate estimation unit 116. Then, the determination unit 130 determines the workpiece including the identified temporary gripping position as the target workpiece. This allows one or more target workpieces to be determined. As a modified example, the determination unit 130 may identify the maximum gripping success rate from among all the gripping success rates output by the success rate estimation unit 116. In this modified example, the determination unit 130 determines the workpiece including the temporary gripping position corresponding to the maximum gripping success rate as the target workpiece.

[0039] When the determination unit 130 determines the target workpiece, it outputs information indicating the target workpiece to the command unit 132. Upon receiving the information indicating the target workpiece, the command unit 132 outputs a control signal to the gripping unit 30. This control signal is for controlling the positioning mechanism 32 to move the gripping unit 30 (hand 31) until the distance between the hand 31 and the target workpiece is a first predetermined distance. The first predetermined distance is a distance that allows for adjustments such as the orientation of the hand 31.

[0040] The control device 50 causes the second camera 12 to capture an image of the target workpiece when the distance between the hand 31 and the target workpiece reaches a first predetermined distance. The determination of whether or not the distance between the hand 31 and the target workpiece has reached the first predetermined distance is achieved by using the second camera 12 as a distance sensor. Note that a different sensor may be used as the distance sensor instead of the second camera 12.

[0041] The second camera 12 generates a second image by capturing an image of the target workpiece. The second acquisition unit 102 acquires the second image from the second camera 12. The second image is output to the second position estimation unit 126 of the second detection unit 124. The second position estimation unit 126 acquires the actual gripping position of the target workpiece based on the second image and the second position model 128. The actual gripping position corresponds to the "first gripping position" in this disclosure.

[0042] The second position model 128 is pre-constructed using machine learning. The second position model 128 is constructed using a neural network. The neural network consists of an input layer, hidden layers, an output layer, and links connecting each layer. When the second image is input to the input layer of the second position model 128, the output layer outputs the actual gripping position of the target workpiece contained in the second image.

[0043] Figure 8 is an example of the second image. In the example in Figure 8, the target workpiece image WS, the gripping part image 31AS of gripping part 31A, and the gripping part image 31BS of gripping part 31B are shown.

[0044] Such a second image is output to the second detection unit 124. Figure 9 shows an example of the concept of the second position model 128. In the example in Figure 9, the second image data H1, H2, ..., Hp are associated with profiles C1, C1, ..., Cp, respectively. Hereinafter, p is an integer of 2 or more. The second image data H1, H2, ..., Hp will be collectively referred to as second image data H, and the profiles C1, C1, ..., Cp will be collectively referred to as profile C.

[0045] The second image data H is image data of a workpiece previously captured by the second camera 12 or a camera equivalent to the second camera 12. In the example in Figure 9, a single workpiece facing various directions, a gripping part image 31AS, and a gripping part image 31BS are shown.

[0046] Profile data is data that defines the main gripping position of each workpiece, oriented in various directions, on a pixel-by-pixel basis. The main gripping position is the gripping position for the gripping unit 30 to properly grip the workpiece.

[0047] The second position model 128 is constructed by manual input from the user. For example, the user causes the second camera 12 or a camera equivalent to the second camera 12 to capture an image of a workpiece and generate second image data. Then, the user displays the image related to the second image data on a display device (not shown) and specifies the actual gripping position of the displayed workpiece.

[0048] In this way, the gripping position specified by the inputter is used as training data, and the gripping position and the second image data are used as learning data to construct the second position model 128.

[0049] Furthermore, as will be described later, the second position model 128 is designed to be automatically updateable. Note that the second position model 128 only needs to be able to detect the actual grasping position from the second image data and does not necessarily need to be constructed using a neural network. For example, it may be constructed using machine learning methods other than neural networks.

[0050] When the second position estimation unit 126 acquires the gripping position, it outputs the gripping position to the command unit 132. The command unit 132 outputs a control signal to the gripping unit 30 for gripping the target workpiece at the gripping position. When the gripping unit 30 receives the control signal, it executes a gripping operation. Here, the gripping operation is the operation of gripping the target workpiece between the gripping parts 31A and 31B at the gripping position. In this embodiment, the gripping operation is the operation of moving the gripping parts 31A and 31B in the closing direction (see Figure 2).

[0051] Furthermore, the gripping unit 30 performs a gripping operation, and with the target workpiece held between the gripping parts 31A and 31B, moves the hand 31 upward (in the Z-axis direction) by a second predetermined distance. Upward includes the direction away from the position of the target workpiece.

[0052] In this case, when the hand 31 moves upward while the target workpiece is held between the gripping parts 31A and 31B, the gripping may fail, for example, if the target workpiece slips off the hand 31. Therefore, in this embodiment, the control device 50 outputs a control signal to execute a gripping operation, and when the gripping unit 30 is moved upward by a second predetermined distance, the second camera 12 takes an image. This image corresponds to the "third image" in this disclosure. The second acquisition unit 102 acquires the third image taken by the second camera 12. This third image is output to the determination unit 120. The determination unit 120 then determines whether or not the target workpiece was gripped based on this third image.

[0053] Figure 10 shows an example of a third image. In the example in Figure 10, the target workpiece image WS, the gripping unit image 31AS, and the gripping unit image 31BS are shown. The example in Figure 10 is a third image captured by the second camera 12 when the gripping unit 30 is moved upward by a second predetermined distance while the target workpiece has been successfully gripped by the gripping unit 30. Therefore, the target workpiece image WS is displayed larger in the third image in Figure 10 than in the second image in Figure 8. Also, in the example in Figure 10, the optical axis of the second camera 12 is shown. As shown in Figure 10, the workpiece is located on the optical axis of the second camera 12 when the gripping units 31A and 31B approach each other along the first predetermined direction.

[0054] Next, the determination method by the determination unit 120 will be described. In this embodiment, the determination unit 120 determines whether or not the target workpiece has been grasped based on the second image and the third image. In this embodiment, the determination unit 120 performs a binarization process on the second image and the third image. The determination unit 120 assigns a first value (for example, 1) to the region of the image (the target workpiece image WS, the grasping unit image 31AS, and the grasping unit image 31BS), and assigns a second value (for example, 0) to the region other than the image. Figure 11 is an example of a binarized second image. Figure 12 is an example of a binarized third image. In the examples of Figures 11 and 12, cross-hatching is added to the image region.

[0055] Furthermore, the determination unit 120 calculates the difference between the number of pixels in the image region (first value region) of the binarized second image and the number of pixels in the image region (first value region) of the binarized third image. Based on this difference, the determination unit 120 determines whether or not the target workpiece has been grasped. For example, the difference is calculated by subtracting the number of pixels in the image region (first value region) of the binarized second image from the number of pixels in the image region (first value region) of the binarized third image. The determination unit 120 then compares the difference with a predetermined threshold. If the grasping is successful, the number of pixels in the target workpiece image WS in the third image will increase. The determination by the determination unit 120 is made based on this phenomenon. If the difference is greater than or equal to the threshold, the determination unit 120 determines that the hand 31 has successfully grasped the target workpiece. On the other hand, if the difference is less than the threshold, the determination unit 120 determines that the hand 31 has failed to grasp the target workpiece.

[0056] The determination method used by the determination unit 120 is not limited to the binarization process described above. For example, the determination unit 120 may determine whether or not the target workpiece was grasped using the third image instead of the second image. For example, the determination unit 120 may determine whether or not the target workpiece was grasped based on the binarized third image. Alternatively, the determination may be made based on the third image and a model constructed by machine learning based on previously collected third images.

[0057] The determination unit 120 outputs the determination result to the command unit 132. The success determination result, indicating that gripping was successful, is a first value (for example, 1), and the failure determination result, indicating that gripping failed, is a second value (for example, 0).

[0058] The command unit 132 executes processing according to the judgment result. When the command unit 132 receives a success judgment result, it outputs a control signal to the gripping unit 30 indicating a continuation operation. The continuation operation is an operation performed by activating the positioning mechanism 32 and the hand 31 in order to move on to the next action when the gripping of the target workpiece by the hand 31 is successful. The continuation operation is, for example, an operation to place the gripped target workpiece in the target area, an assembly operation using the gripped target workpiece, and an operation related to fitting.

[0059] On the other hand, the command unit 132 outputs a control signal to the gripping unit 30 indicating an interruption operation when it receives a failure determination result. The interruption operation is an operation that interrupts the next action when the gripping of the target workpiece by the hand 31 fails. The intermediate operation is, for example, an operation that moves the gripping unit 30 to the initial position (the position when the first image was acquired).

[0060] Furthermore, the determination unit 120 also outputs the determination result to the determination data storage unit 118. The determination data storage unit 118 stores the determination data B1 to Bm shown in Figure 7. When the determination data storage unit 118 receives a determination result from the determination unit 120, it updates the determination data by adding data based on the determination result.

[0061] Figure 13 shows an example of updating the judgment data in the judgment data storage unit 118. In the example in Figure 13, it is shown that new judgment data Bm+1 has been added to the existing judgment data B1 to Bm (see also Figure 7). The new judgment data Bm+1 is information generated by the judgment unit 120, etc., based on the judgment result from the judgment unit 120.

[0062] When the judgment data in the judgment data storage unit 118 is updated, the construction unit 112 updates the gripping success rate model 114 to reflect the updated judgment data (i.e., the judgment data after the addition of the judgment data Bm+1 in Figure 13). Note that updating the model (such as the gripping success rate model 114) includes, for example, a process to update the parameters defined in the model. Updating the gripping success rate model 114 can improve the accuracy of the gripping success rate estimation by the success rate estimation unit 116.

[0063] Furthermore, the construction unit 112 may update the first position model 106 and the second position model 128 to reflect the updated judgment data (i.e., the judgment data after the addition of the judgment data Bm+1 in Figure 13). Updating the first position model 106 can improve the accuracy of the provisional gripping position estimation by the first position estimation unit 108. Also, updating the second position model 128 can improve the accuracy of the final gripping position estimation by the second position estimation unit 126.

[0064] Furthermore, the gripping device 100 may update at least one of the first position model 106, the second position model 128, and the gripping success rate model 114. Also, the gripping device 100 may update at least one of the above models when a predetermined number of judgment data have been accumulated in the judgment data storage unit 118.

[0065] [flowchart] Figure 14 is a flowchart of the processing of the gripping device 100. This flowchart starts when the user performs the start operation described above. In step S2, the gripping device 100 moves the gripping unit 30 to its initial position. Next, in step S4, the gripping device 100 acquires a first image from the first camera 11. Next, in step S6, the gripping device 100 acquires the temporary gripping position of each of at least one workpiece placed on the workbench 60 based on the first image acquired in step S4.

[0066] Next, in step S8, the gripping device 100 obtains the gripping success rate for at least one workpiece based on the respective temporary gripping positions obtained in step S6. Next, in step S10, it is determined whether a gripping success rate higher than a predetermined threshold was obtained in step S8. If a gripping success rate higher than the threshold is obtained (YES in step S10), the process proceeds to step S12. On the other hand, if a gripping success rate higher than the threshold is not obtained, that is, if all obtained gripping success rates are below the threshold (NO in step S10), the gripping operation is terminated. If NO is determined in step S10, it means that there are no workpieces that the gripping unit 30 can grip. Therefore, for example, the gripping device 100 may issue a notification to the user indicating that "there are no workpieces that can be gripped."

[0067] In step S12, the gripping device 100 determines the target workpiece from at least one workpiece based on each temporary gripping position. Next, in step S14, the gripping device 100 moves the gripping unit 30 until the distance between the hand 31 and the target workpiece becomes a first predetermined distance.

[0068] Next, in step S16, the gripping device 100 acquires a second image. Next, in step S18, the gripping device 100 acquires the final gripping position based on the second position model 128 and the second image. Next, in step S20, the gripping device 100 performs a gripping operation (an operation in which the gripping parts 31A and 31B grip the target workpiece at the final gripping position) by outputting a control signal.

[0069] Next, in step S22, the gripping device 100 raises the hand 31 by a second predetermined distance. Next, in step S24, the gripping device 100 acquires a third image (see Figure 10). Next, in step S26, the gripping device 100 determines whether the gripping was successful or not (see Figures 11 and 12).

[0070] Next, in step S28, the gripping device 100 stores determination data (determination result) indicating whether or not the gripping was successful in the determination data storage unit 118. Next, in step S30, the gripping device 100 determines whether or not a predetermined number of determination data have been accumulated in the determination data storage unit 118.

[0071] If the result in step S30 is YES, the process proceeds to step S32. If the result in step S30 is NO, the process proceeds to step S34. In step S32, the gripping device 100 updates the first position model 106, the second position model 128, and the gripping success rate model 114. In step S34, the gripping device 100 determines whether or not the target workpiece was successfully gripped. This determination is made, for example, based on the determination data stored in step S28. If the determination data indicates that the gripping was successful, the result in step S34 is YES. On the other hand, if the determination data indicates that the gripping failed, the result in step S34 is NO.

[0072] Furthermore, if the number of target workpieces determined in step S12 is multiple, and the result in step S34 is YES, the gripping device 100 repeats the process from step S14 to step S34 until all target workpieces have been gripped.

[0073] Furthermore, if the gripping device 100 determines YES in step S34, the process returns to S2, for example, and is repeated until the workpiece is no longer included in the first image captured in step S4.

[0074] As described above, in this embodiment, the gripping device 100 acquires the success rate of gripping each of at least one workpieces placed on the workbench 60 based on the first image captured by the first camera (step S8). The gripping device 100 also determines a target workpiece from at least one workpiece based on the success rate of each gripping (step S12). The gripping device 100 moves the gripping unit 30 until the distance between the hand 31 and the target workpiece is a first predetermined distance (step S14). After moving the gripping unit 30, the gripping device 100 acquires a second image by having the second camera 12 capture an image (step S16). The gripping device 100 also acquires the final gripping position of the target workpiece based on the second image (step S18). The gripping device 100 then outputs a control signal to cause the gripping unit 30 to grip the target workpiece at the final gripping position (step S20).

[0075] As a comparative example of a gripping device, one could consider a "gripping device that performs the process of determining a target workpiece from at least one workpiece and the process of determining the gripping position of the target workpiece in a single step." However, with such a comparative example of a gripping device, it was difficult to improve the success rate of gripping due to reasons such as the need for precise calculations. In contrast, the gripping device 100 determines a target workpiece from at least one workpiece in the first step, and determines the actual gripping position of the target workpiece in the second step. Therefore, since it is not necessary to determine the target workpiece and its gripping position in a single step, the success rate of gripping the workpiece can be improved as a result.

[0076] Furthermore, the gripping device 100 outputs a control signal to cause the gripping unit 30 to grip the target workpiece (step S20). After that, the gripping device 100 raises the hand 31 by a second predetermined distance (step S20) and then acquires a third image by having the second camera 12 capture the image. The gripping device 100 then determines whether or not the gripping unit has been able to grip the target workpiece based on the third image. In conventional gripping devices, it was necessary to move the hand into the imaging range of the overhead camera in order to determine whether or not the target workpiece had been gripped. Therefore, the cycle time could be long. However, in this embodiment, it is possible to determine whether or not the target workpiece has been gripped based on the third image captured by the second camera positioned on the gripping unit 30. Therefore, since it is not necessary to perform a process such as moving the hand into the imaging range of the overhead camera, the cycle time can be shortened compared to conventional gripping devices.

[0077] Furthermore, in this embodiment, the gripping device 100 can automatically update at least one of the first position model 106, the second position model 128, and the gripping success rate model 114. Therefore, it is possible to improve the workpiece gripping success rate while reducing the processing burden on the user.

[0078] Furthermore, in a gripping device where the control device 50 does not acquire a second image of the hand 31 gripping the target workpiece, it was necessary to calculate an offset to correct for misalignment of the optical axis of the second camera 12 in order for the second camera 12 to image the target workpiece. Therefore, such a gripping device may lead to increased calculation processing and increased cycle time. In contrast, the second camera 12 in this embodiment is fixed so that the control device 50 acquires a second image of the hand 31 gripping the target workpiece (see Figure 2, etc.). Therefore, the above-mentioned calculation processing and cycle time can be reduced.

[0079] Furthermore, as a comparative example of a gripping device, one can consider a "gripping device in which the optical axis of the second camera and the direction of movement (X-axis direction) of the multiple gripping parts (two gripping parts 31A, 31B) are not perpendicular." However, in such a comparative example of a gripping device, it may not be possible to properly capture the second and third images. On the other hand, in the gripping device 100 of this embodiment, the optical axis of the second camera 12 is perpendicular to the direction of movement (X-axis direction) of the two gripping parts 31A, 31B). Also, when the multiple gripping parts approach each other along the first predetermined direction, the workpiece lies on the optical axis of the second camera 12 (see Figure 10). Therefore, when the multiple gripping parts approach each other along the first predetermined direction, the workpiece can be properly captured as the third image.

[0080] [Other hand configurations] Figures 15 and 16 show another example of hand 31, hand 31a. Compared to hand 31 in Figure 2, hand 31a replaces the opening / closing section 31E with opening / closing sections 31F and 31G. Also, in the hand 31 examples in Figures 2 and 3, the gripping sections 31A and 31B are L-shaped in the YZ plane, whereas in hand 31a in Figures 15 and 16, the gripping sections 31A and 31B are L-shaped in the XZ plane. Figure 17 is a third image when hand 31a is used. In the example in Figure 17, the target workpiece image WS, the opening / closing section image 31FS of opening / closing section 31F, and the opening / closing section image 31GS of opening / closing section 31G are displayed. The optical axis of the second camera 12 is also indicated. In the example shown in Figure 17, the workpiece is located on the optical axis of the second camera when the gripping parts 31A and 31B approach each other along the first predetermined direction.

[0081] Furthermore, in the above example, the hand 31 and hand 31a were described as being of the so-called chuck type. However, the hand may also be of the so-called suction type. Figure 18 is an example of a hand 31b that employs a suction type. In the example of Figure 18, the second camera 12 is fixed so as to be able to image the target workpiece W that is grasped (suctioned) by the hand 31b. Figure 19 is a third image when hand 31b is used. In the example of Figure 19, the target workpiece image WS and the hand image 31bS of hand 31b are displayed. In the example of Figure 19, the optical axis of the second camera 12 is indicated. In the example of Figure 19, the hand 31b is located on the optical axis of the second camera. Note that the target workpiece may be located on the optical axis of the second camera.

[0082] Figure 20 shows another embodiment of the hand 31b. In the example of Figure 20, the gripping device 100 has an illumination device 20. Figure 21 is a plan view from the optical axis of the second camera 12. The control device 50 illuminates the imaging range of the second camera 12. As a result, the target workpiece gripped by the hand 31b can be illuminated, and the second camera 12 can generate clearer second and third images. Therefore, the gripping device 100 can improve the success rate of gripping the workpiece. Note that the illumination device 20 is not limited to Figures 20 and 21 and may be mounted in other embodiments. Also, the illumination device 20 may be mounted on the hand 31 or the hand 31a.

[0083] [Note] (Section 1) The gripping device of the present disclosure comprises a first camera, a gripping unit, a second camera, and a control device. The first camera images at least one workpiece. The gripping unit has a hand for gripping the workpiece. The second camera is positioned on the gripping unit and images the hand. The control device controls the first camera, the second camera, and the gripping unit. Based on a first image captured by the first camera, the control device obtains the success rate of gripping each of the at least one workpiece. Based on the success rate of gripping each, the control device determines a target workpiece from the at least one workpiece. The control device moves the gripping unit until the distance between the hand and the target workpiece is a first predetermined distance. After moving the gripping unit, the control device obtains a second image by having the second camera capture an image. Based on the second image, the control device obtains a first gripping position of the target workpiece. The control device then outputs a control signal for the gripping unit to grip the target workpiece at the first gripping position.

[0084] (Clause 2) The gripping device described in Clause 1, wherein the control device outputs a control signal to move the gripping unit upward by a second predetermined distance, and acquires a third image by causing the second camera to take an image. The control device also determines whether the gripping unit was able to grip the target workpiece based on the third image.

[0085] (Clause 3) The gripping device described in Clause 2, wherein the control device has a first estimation model and a second estimation model. The control device acquires the second gripping position of each of at least one workpiece based on the first estimation model and the first image. The control device acquires the success rate of each gripping based on the second estimation model and each second gripping position. The control device then updates at least one of the first estimation model and the second estimation model based on the determination result of whether or not the target workpiece was gripped.

[0086] (Clause 4) The gripping device described in paragraph 2 or 3, wherein the control device has a third estimation model. The control device acquires a first gripping position based on the third estimation model and the second image. The control device updates the third estimation model based on the determination result of whether or not the target workpiece was gripped.

[0087] (Clause 5) A gripping device as described in any one of paragraphs 1 to 4, wherein the second camera is fixed so that the control device acquires a second image of the hand gripping the target workpiece.

[0088] (Paragraph 6) A gripping device according to any one of paragraphs 1 to 5, wherein the gripping device further comprises an illumination device for illuminating the imaging range of the second camera.

[0089] (Clause 7) A gripping device according to any one of paragraphs 1 to 6, wherein the gripping unit has a plurality of gripping parts. The control device outputs a control signal so that the plurality of gripping parts move closer to each other along a first predetermined direction. The optical axis of the second camera is perpendicular to the first predetermined direction. The workpiece when the plurality of gripping parts move closer to each other along the first predetermined direction is located on the optical axis of the second camera.

[0090] (Section 8) The control method of the present disclosure is a control method for a gripping device. The gripping device comprises a first camera, a gripping unit, and a second camera. The first camera images at least one workpiece. The gripping unit has a hand for gripping workpieces. The second camera is positioned on the gripping unit and images the hand. The control method comprises obtaining the success rate of gripping each of at least one workpiece based on a first image captured by the first camera. The control method also comprises determining a target workpiece from at least one workpiece based on the success rate of gripping each workpiece. The control method also comprises moving the gripping unit until the distance between the hand and the target workpiece is a first predetermined distance. The control method also comprises obtaining a second image by having the second camera capture an image after moving the gripping unit. The control method also comprises obtaining a first gripping position of the target workpiece based on the second image. The control method also comprises outputting a control signal to cause the gripping unit to grip the target workpiece at the first gripping position.

[0091] Although embodiments of the present invention have been described above, it is possible to modify these embodiments in various ways. Furthermore, the scope of the present invention is not limited to the embodiments described above. The scope of the present invention is indicated by the claims and is intended to include all modifications within the meaning and scope equivalent to the claims. [Explanation of Symbols]

[0092] 11 First camera, 12 Second camera, 15 Support unit, 20 Lighting device, 30 Gripping unit, 31, 31a, 31b Hand, 31A, 31B Gripping part, 31AS, 31BS Gripping part image, 31C, 31D Fixing part, 31E, 31F, 31G Opening / closing part, 31FS, 31GS Opening / closing part image, 31bS Hand image, 32 Positioning mechanism, 32A Tip part, 50 Control device, 54 Memory, 60 Workbench, 100 Gripping device, 101 First acquisition unit, 102 Second acquisition unit, 104 First detection unit, 106 First position model, 108 First position estimation unit, 110 Output unit, 112 Construction unit, 114 Gripping success rate model, 116 Success rate estimation unit, 118 Judgment data storage unit, 120 Judgment unit, 124 Second detection unit, 126 second position estimation unit, 128 second position model, 130 determination unit, 132 command unit.

Claims

1. A first camera that images at least one workpiece, A gripping unit having a hand for gripping a workpiece, A second camera is positioned in the gripping unit and captures images of the hand, The system comprises the first camera, the second camera, and a control device for controlling the gripping unit, The control device is Based on the first image captured by the first camera, the success rate of gripping each of the at least one workpiece is obtained. Based on the respective gripping success rates, the target workpiece is selected from the at least one workpiece. The gripping unit is moved until the distance between the hand and the target workpiece becomes a first predetermined distance. After moving the gripping unit, a second image is acquired by having the second camera capture the image. Based on the second image, the first gripping position of the target workpiece is obtained. A control signal is output to cause the gripping unit to grip the target workpiece at the first gripping position. When the aforementioned control signal is output and the gripping unit is moved upward by a second predetermined distance, a third image is acquired by causing the second camera to capture the image. A gripping device that determines whether the gripping unit was able to grip the target workpiece based on the third image.

2. The control device has a first estimation model and a second estimation model, The control device is Based on the first estimation model and the first image, the second gripping position of each of the at least one workpiece is obtained. Based on the second estimation model and the respective second gripping positions, the respective gripping success rates are obtained. The gripping device according to claim 1, wherein at least one of the first estimation model and the second estimation model is updated based on the determination result of whether or not the target workpiece was gripped.

3. The control device has a third estimation model, The control device is Based on the third estimation model and the second image, the first gripping position is obtained. The gripping device according to claim 1 or claim 2, which updates the third estimation model based on the determination result of whether or not the target workpiece was gripped.

4. The gripping device according to claim 1 or 2, wherein the second camera is fixed so that the control device can acquire the second image of the hand gripping the target workpiece.

5. The gripping device according to claim 1 or claim 2, further comprising an illumination device for illuminating the imaging range of the second camera.

6. The gripping unit has a plurality of gripping parts, When the control device outputs the control signal, the plurality of gripping parts move closer to each other along the first predetermined direction. The optical axis of the second camera is perpendicular to the first predetermined direction, The gripping device according to claim 1 or claim 2, wherein the workpiece is located on the optical axis of the second camera when the plurality of gripping portions approach each other along a first predetermined direction.

7. A method for controlling a gripping device, The gripping device is A first camera that images at least one workpiece, A gripping unit having a hand for gripping a workpiece, The gripping unit is equipped with a second camera that captures images of the hand, The control method described above is Based on the first image captured by the first camera, the success rate of gripping each of the at least one workpiece is obtained, Based on the respective gripping success rates, the target workpiece is determined from the at least one workpiece, The gripping unit is moved until the distance between the hand and the target workpiece becomes a first predetermined distance. After moving the gripping unit, a second image is acquired by having the second camera capture the image. Based on the second image, the first gripping position of the target workpiece is obtained, Outputting a control signal to cause the gripping unit to grip the target workpiece at the first gripping position, When the aforementioned control signal is output and the gripping unit is moved upward by a second predetermined distance, a third image is acquired by causing the second camera to capture an image. A control method comprising determining whether the gripping unit was able to grip the target workpiece based on the third image.

Images