Control device and program

Abstract

A control device according to the present disclosure comprises a storage unit which stores data relating to a plurality of holding patterns for holding a workpiece through use of an end effector of a robot, a selection unit which selects one holding pattern for the workpiece from among the plurality of holding patterns, a control unit which controls the robot on the basis of the selected one holding pattern, a creation unit which creates a plurality of holding pattern candidates on the basis of the holding pattern, a reception unit which receives a user instruction for adding at least one of the created holding pattern candidates as the holding pattern, and a storage control unit which stores the received holding pattern candidate, as the holding pattern, in the storage unit.

Description

Control device and program

[0001] The present disclosure relates to a control device and a program.

[0002] A technique for bulk picking is known in which a robot picks up a plurality of workpieces that are randomly arranged at different intervals and in different orientations in a container or other vessel, so-called bulk picking (see, for example, Patent Document 1). For example, in bulk picking, candidate positions for gripping the workpieces with a hand mounted on the robot are taught and set in advance to the robot.

[0003] Japanese Patent Application Laid-Open No. 2019-028775

[0004] Because a robot can only pick up a workpiece at a set gripping position, there are cases where the robot cannot pick up a workpiece from a container due to interference between the robot's hand and the container or the workpiece. To address such cases, a user must observe the position and orientation of the workpiece that cannot be picked up by the hand and remains in the container, and teach the gripping position of the workpiece that can be picked up by the hand. Teaching the gripping position requires the user to use an operating device such as a pendant while considering what gripping position should be taught so that the workpiece can be picked up by the hand. This is a burdensome task for users who are unfamiliar with setting up bulk picking. Therefore, there is a need for a technology that reduces the effort required to teach the gripping position when picking up bulk items.

[0005] The control device according to the present disclosure includes a memory unit that stores data regarding a plurality of holding patterns for holding a workpiece with the end effector of the robot, in order to control a robot that performs the task of removing bulk workpieces, a selection unit that selects one of the plurality of holding patterns for the workpiece, a control unit that controls the robot based on the selected holding pattern, a creation unit that creates a plurality of holding pattern candidates based on the holding patterns, a reception unit that receives user instructions to add at least one of the created holding pattern candidates as a holding pattern, and a memory control unit that stores the received holding pattern candidate in the memory unit as a holding pattern.

[0006] FIG. 1 is a diagram showing a robot system including a control device according to a first embodiment. FIG. 2 is a hardware configuration diagram of the control device of FIG. 1. FIG. 3 is a functional block diagram of the control device of FIG. 1. FIG. 4 is a diagram showing an example of a holding pattern management table stored in the storage unit of FIG. 3. FIG. 5 is a supplementary diagram for explaining the holding patterns of FIG. 4. FIG. 6 is a supplementary diagram for explaining the process of creating candidates for the holding patterns of FIG. 4. FIG. 7 is a diagram showing an example of a reception screen created by the screen creation unit of FIG. 3. FIG. 8 is a flowchart showing an example of a procedure for a workpiece removal operation performed by the control device according to the first embodiment. FIG. 9 is a flowchart showing an example of a procedure for step S22 of FIG. 8. FIG. 10 is a diagram showing an example of a reception screen updated after clicking the Create button of FIG. 7. FIG. 11 is a diagram showing an example of a reception screen updated after clicking the Add button of FIG. 10. FIG. 12 is a diagram showing another example of a reception screen updated after clicking the Create button of FIG. 7. FIG. 13 is a diagram showing an example of a reception screen updated after clicking the Execute button of FIG. 11. FIG. 14 is a functional block diagram of a control device according to a second embodiment. Fig. 15 shows an example of a reception screen presented to a user by a control device according to the second embodiment. Fig. 16 is a diagram showing the reception screen when a cursor is aligned with interference information in Fig. 15. Fig. 17 is a diagram showing the reception screen when an adjustment button is clicked in Fig. 15.

[0007] The control devices according to the first and second embodiments will be described below with reference to the drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0008] In this embodiment, the terms are defined as follows: Holding pattern: A parameter set for a robot to perform a workpiece picking operation. A holding pattern typically includes a position (holding position) at which the robot's hand (end effector) holds a workpiece and a posture (holding posture) at which the workpiece is held. Of course, a holding pattern may represent only one of the holding position and the holding posture. Typically, multiple holding patterns are prepared for bulk picking. The control device selects one of the multiple holding patterns for one workpiece to be picked, and controls the robot based on the selected holding pattern. This allows the workpiece to be picked from a container, etc.

[0009] Candidate for holding pattern: A parameter that is a candidate to be registered as a holding pattern in the robot or the control device that controls the robot. Candidate for holding pattern is created based on an existing holding pattern.

[0010] In this embodiment, "holding" is a concept that includes suction, gripping, etc. In other words, when a hand having a function of gripping a workpiece is used, the holding pattern (holding position, holding posture) may be rephrased as a gripping pattern (gripping position, gripping posture). Similarly, when a hand having a function of suctioning a workpiece is used, the holding pattern (holding position, holding posture) may be rephrased as a suction pattern (suction position, suction posture).

[0011] The control device of the present disclosure is a computer device having the following functions: creating holding pattern candidates based on a plurality of pre-registered holding patterns; receiving a user instruction to additionally register at least one holding pattern candidate from the created holding pattern candidates as a holding pattern; and storing the received at least one holding pattern candidate as a holding pattern. (First Embodiment) As shown in FIG. 1 , a robot system 1 including a control device 2 according to a first embodiment includes a robot 10 equipped with a robot hand 11 as an end effector for removing workpieces W bulk loaded in a container 15; a three-dimensional vision camera 13 positioned to overlook the opening of the container 15 that stores the workpieces W and captures images of the container 15; and a control device 2 that controls the robot 10 and the three-dimensional vision camera 13. The three-dimensional vision camera 13 generates image data of the workpieces W stored in the container 15. The image data is also referred to as point cloud data.

[0012] As shown in FIG. 2, the control device 2 is configured by connecting hardware such as an operation device 6, a display device 7, a communication device 8, and a storage device 9 to a processor 5 such as a CPU.

[0013] The operation device 6 is realized by a keyboard, a mouse, a jog, etc. The operation device 6 may be realized by a touch panel that also serves as the display device 7, or may be realized by a dedicated pendant for the robot 10. A user can input various information to the control device 2 via the operation device 6. The display device 7 is realized by an LCD, etc., and displays various screens according to the control of the processor 5. The communication device 8 is realized by a communication module that complies with any communication standard, and transmits and receives various data between the robot 10 and the 3D vision camera 13 according to the control of the processor 5. The storage device 9 is realized by an HDD, SSD, etc. A control program and an assistance program are stored in the storage device 9. The assistance program may be configured as part of the control program, or may be executed in conjunction with the control program.

[0014] As shown in Figure 3, the control program stored in the storage device 9 is executed by the processor 5 together with the assistance program, so that the control device 2 functions as a reception unit 21, a display unit 22, an acquisition unit 23, a storage unit 24, a robot control unit 25, a work detection unit 26, a target work determination unit 27, a judgment unit 28, a holding pattern selection unit 29, a holding pattern candidate creation unit 30, a storage control unit 31, and a screen creation unit 32.

[0015] The reception unit 21 receives user operations via the operation device 6. The reception unit 21 is realized by the functions of the operation device 6 shown in FIG. 2 . Specifically, the reception unit 21 receives user operations on a reception screen displayed on the display device 7. User operations on the reception screen include clicking the Create button, clicking the Execute button, clicking the Add button, selecting at least one retention pattern candidate from multiple retention pattern candidates to be stored in the memory unit 24 as a retention pattern, and inputting the priority of at least one retention pattern candidate to be stored in the memory unit 24. The display unit 22 displays the reception screen created by the screen creation unit 32. The display unit 22 is realized by the functions of the display device 7 shown in FIG. 2 . The acquisition unit 23 acquires image data of the container 15 from the 3D vision camera 13. The acquisition unit 23 is realized by the functions of the communication device 8 shown in FIG. 2 .

[0016] The memory unit 24 stores various data necessary for the robot 10 to execute bulk picking. The memory unit 24 is realized by the functions of the storage device 9 shown in FIG. 2. The various data necessary for controlling the operation of the robot 10 include an operation sequence, setting data, and a holding pattern management table. The operation sequence describes the procedure for causing the robot 10 to execute bulk picking. The setting data includes the operation speed, operation format, interpolation format, etc. of the robot 10. The holding pattern management table stores the holding patterns of the robot hand 11 with respect to the workpiece W in table format. Details of the holding pattern management table will be described later. Note that the holding patterns of the robot hand 11 with respect to the workpiece W may be stored in the memory unit 24 in another format other than the table format.

[0017] The storage unit 24 stores various data necessary for support processing to support the user in adding a holding pattern. The various data necessary for support processing includes various data necessary for screen creation by the screen creation unit 32 and data necessary for judgment processing by the judgment unit 28. The various data necessary for screen creation includes a screen format, text data, image data, a basic model of a schematic diagram, etc. The data necessary for judgment processing includes data necessary for executing a simulation in which the robot hand 11 holds the workpiece W accommodated in the container 15, such as data of a three-dimensional model of the robot 10 including the robot hand 11, data of a three-dimensional model of the workpiece W, and data of a three-dimensional model of the container 15.

[0018] The robot control unit 25 controls the robot 10 and the three-dimensional vision camera 13 according to an operation sequence to cause the robot 10 to perform an operation of picking up the workpieces W bulk stacked in the container 15. In the procedure for holding the target workpieces W, the robot control unit 25 controls the robot 10 based on a holding pattern selected by a holding pattern selection unit 29 described below.

[0019] The workpiece detection unit 26 detects the workpiece W inside the container 15 based on the image data acquired from the 3D vision camera 13. For example, the detection of the workpiece W can employ known image processing methods such as template matching using pre-registered template data, 3D matching using shape data of a 3D model of the workpiece W stored in the storage unit 24, extracting the interior area of ​​the container 15 and performing pixel threshold processing on the extracted interior area, etc.

[0020] The target work determination unit 27 determines the work W to be held (referred to as the target work W) based on the image data acquired from the three-dimensional vision camera 13. For example, the target work determination unit 27 randomly determines the target work W from the work W whose entirety is detected in the image data, in other words, the work W whose entirety is exposed on the opening side of the container 15.

[0021] The determination unit 28 determines the possibility of holding the target workpiece W by the holding pattern (candidate holding pattern) based on the imaging data acquired from the 3D vision camera 13. Specifically, the determination unit 28 calculates the position and posture of the target workpiece W in the container 15 and the positions and postures of the other workpieces W based on the imaging data. Then, the determination unit 28 calculates the holding position and holding posture of the target workpiece W based on the holding pattern, and calculates the possibility of interference of the robot hand 11 with the container 15 and the other workpieces W when the robot hand 11 is placed in the holding posture at the holding position of the target workpiece W. Then, the determination unit 28 calculates the holding possibility based on the possibility of interference.

[0022] For example, the possibility of interference of the robot hand 11 with the container 15 or other workpieces W is calculated as follows: That is, if the area corresponding to the robot hand 11 overlaps with the areas corresponding to the container 15 or other workpieces W, the possibility of interference is calculated as 100%, and if the area corresponding to the robot hand 11 does not overlap with the areas corresponding to the container 15 or other workpieces W and the two areas are separated by a predetermined distance, the possibility of interference is calculated as 0%. If the area corresponding to the robot hand 11 does not overlap with the areas corresponding to the container 15 or other workpieces W but the two areas are close to each other and less than a predetermined distance apart, the possibility of interference is calculated according to the distance.

[0023] The determination unit 28 calculates the holding possibility by subtracting the interference possibility from 100%. That is, an interference possibility of 100% is expressed as a holding possibility of 0%, and an interference possibility of 0% is expressed as a holding possibility of 100%. The determination unit 28 compares the holding possibility with a preset threshold value to determine whether or not the target workpiece W can be held by each holding pattern.

[0024] The holding pattern selection unit 29 selects one holding pattern from the plurality of holding patterns to be used for holding the target workpiece W. Specifically, the holding pattern selection unit 29 selects from the plurality of holding patterns a holding pattern that has been determined by the determination unit 28 to have a high possibility of holding the target workpiece W. When a priority order is set for the holding patterns, the holding pattern selection process is performed by verifying the holding patterns in order of priority.

[0025] The hold pattern candidate creating unit 30 creates hold pattern candidates based on existing hold patterns. Details of the process of creating hold pattern candidates will be described later.

[0026] The storage control unit 31 stores the candidate holding pattern selected by the user operation as a holding pattern in the storage unit 24. As a result, the accepted candidate holding pattern is added to the existing holding patterns.

[0027] The screen creation unit 32 creates a reception screen in accordance with a predetermined format. The reception screen will be described in detail later.

[0028] The holding pattern management table will be described below with reference to Fig. 4. Fig. 4 shows an example of the holding pattern management table. As shown in Fig. 4, in the holding pattern management table, each of a plurality of holding patterns is managed together with registration date and time information and priority. The holding pattern includes parameters related to the holding position and parameters related to the holding posture. Typically, the parameters related to the holding position and the parameters related to the holding posture are expressed in a workpiece coordinate system. The workpiece coordinate system is a coordinate system defined by the target workpiece W.

[0029] The holding patterns will be described below with reference to FIG. 5 . Here, the holding pattern H1 managed in the holding pattern management table of FIG. 4 will be used as an example. As shown in FIG. 5 , the workpiece W is assumed to be a cylindrical body. The workpiece coordinate system defined by the workpiece W and the hand coordinate system defined by the robot hand 11 are defined as follows: the center position of the cylinder is defined as the origin Ow of the workpiece coordinate system, an axis parallel to the center line of the cylinder is defined as the Xw axis, and two axes perpendicular to the Xw axis and perpendicular to each other are defined as the Yw axis and the Zw axis. The grip center position of the pair of fingers is defined as the origin Oh of the hand coordinate system, an axis along the opening and closing direction of the pair of fingers is defined as the Yh axis, an axis along the center line of the robot hand 11 is defined as the Zh axis, and an axis perpendicular to the Yh axis and the Zh axis is defined as the Xh axis.

[0030] When the workpiece coordinate system and the hand coordinate system are defined as above, as shown in Figure 5, holding pattern H1 ((Xw, Yw, Zw, Ww, Pw, Rw) = (10 mm, 0, 0, 0, 0, 0)) means that the workpiece W is held at a position shifted 10 mm along the central axis (Xw axis) from the center position (Ow) of the workpiece W, in an attitude in which the robot hand 11 is not tilted relative to the workpiece W.

[0031] In addition, in cases where the axes of the workpiece coordinate system cannot be defined solely by the characteristics of the workpiece W, such as a workpiece W having a line-symmetric or point-symmetric shape such as a cylinder, sphere, or rectangle, at least one of the three axes defining the workpiece coordinate system may be defined based on another fixed coordinate system such as a container coordinate system (Xc, Yc, Zc). Furthermore, the holding pattern of the workpiece W may be expressed in another coordinate system such as a container coordinate system instead of the workpiece coordinate system.

[0032] The process of generating holding pattern candidates by the holding pattern candidate generation unit 30 will be described below with reference to FIG. 6 . Typically, the holding pattern candidate generation unit 30 generates holding pattern candidates based on the holding pattern, a 3D model of the container 15, a 3D model of the target workpiece W, and a 3D model of the robot hand 11. Specifically, the holding pattern candidate generation unit 30 extracts multiple holding pattern candidates by searching a predetermined range from the holding pattern, and generates at least one holding pattern candidate from the extracted multiple holding pattern candidates. The predetermined range may be set according to a user specification received via the reception unit 21, or may be automatically set according to the target workpiece W. As shown in FIG. 6 , for example, when the length of the target workpiece W is Lw, the predetermined range searched for holding pattern candidates is maximized by the range in which the holding position Xw is less than ±Lw / 2. The narrower the search range, the faster the calculation processing speed.

[0033] An example of creating a candidate holding pattern based on the holding pattern H1 managed in the holding pattern management table of Fig. 4 will be described below. As shown in Fig. 6, the candidate holding pattern creating unit 30 searches the length Lw of the target workpiece W as a range, extracts multiple candidate holding patterns (H11, H12, ..., H20) by shifting the holding pattern H1 by a unit length ΔL along the Xw axis direction, and then extracts and creates one candidate holding pattern (H18) from the extracted multiple candidate holding patterns (H11, H12, ..., H20) that can hold the target workpiece W (i.e., does not interfere with surrounding objects) and is closest to the existing holding pattern H1. Note that whether the candidate holding pattern can hold the target workpiece W, in other words, whether the robot hand 11 operating according to the candidate holding pattern will interfere with surrounding objects, can be determined by a simulation using a three-dimensional model of the container 15, a three-dimensional model of the target workpiece W, and a three-dimensional model of the robot hand 11, similar to the processing of the determining unit 28.

[0034] The reception screen will be described below with reference to FIG. 7 . FIG. 7 shows an example of the reception screen displayed on the display device 7. As shown in FIG. 7 , the reception screen 100 has a first area 110, a second area 113, and a third area 115. The first area 110 displays photographic data related to the container 15′ and an execute button 111. The second area 113 displays holding patterns stored in the memory unit 24 along with a priority order 121. For example, the holding pattern is displayed as a schematic diagram of the workpiece W and the robot hand 11 so that the holding position and holding posture relative to the workpiece W can be seen. The third area 115 displays a create button 117. Although not shown in FIG. 7 , the third area displays multiple schematic diagrams each representing a multiple holding pattern candidate (see FIGS. 10 and 11 ).

[0035] The bulk picking operation by the robot 10 will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of a control procedure for the robot 10 by the control device 2 when the robot 10 is made to pick up bulk items.

[0036] As shown in FIG. 8 , when the control program is executed, the control device 2 controls the 3D vision camera 13 to acquire image data of the workpieces W stored in the container 15 from the 3D vision camera 13 (S11), and determines whether or not the workpieces W are present inside the container 15 based on the acquired image data (S12). If a workpiece W is present inside the container 15 (S13; Yes), the control device 2 determines the target workpieces W to be held based on the image data (S14), and determines the holding feasibility of the determined target workpieces W based on multiple holding patterns (S15). The determination of the holding feasibility of the target workpieces W using multiple holding patterns is performed in order according to the priority of the holding patterns. Note that by assigning priorities, the processing of step S15 is terminated when a holding pattern with a high possibility of holding the target workpieces W is found. Therefore, by giving a high priority to frequently used holding patterns, the processing speed of step S15 can be suppressed.

[0037] When the possibility of holding the target workpiece W is equal to or greater than a predetermined value, in other words, when the target workpiece W can be held in any of the multiple holding patterns (S16; No), the control device 2 selects a holding pattern that can hold the target workpiece W from the multiple holding patterns (S17) and controls the robot 10 according to the selected holding pattern (S18). As a result of the processing in step S18, the target workpiece W is removed from the container 15. On the other hand, when the possibility of holding the target workpiece W is less than the predetermined value, in other words, when the target workpiece W cannot be held in any of the multiple holding patterns (S16; Yes), the control device 2 determines whether or not there are other unverified workpieces W based on the image data (S19). Here, the other unverified workpieces W are workpieces W for which the determination processing in step S15 has not been performed.

[0038] When there are other unverified workpieces W, in other words, when there is still a possibility that the workpieces W remaining in the container 15 can be held by the existing holding pattern (S20; Yes), the control device 2 changes the target workpiece W to the unverified workpiece W (S21) and returns the process to step S15. On the other hand, when there are no other unverified workpieces W, in other words, when the workpieces W remaining in the container 15 cannot be held by the existing holding pattern (S20; No), the control device 2 executes additional registration processing of a new holding pattern (S22) and returns the process to step S15.

[0039] The series of processes from step S11 to step S22 is repeatedly executed until all of the workpieces W accommodated in the container 15 are removed and there are no more workpieces W inside the container 15. Based on the fact that there are no more workpieces W inside the container 15, the control of the robot 10 by the control device 2 is terminated, and the bulk removal by the robot 10 is completed (S13; No).

[0040] The user assistance process by the control device 2 according to the first embodiment will be described below with reference to Figures 7, 9, 10, and 11. Figure 9 is a flowchart showing an example of the procedure of step S22 in Figure 8. Figure 10 shows the reception screen updated when the Create button is clicked. Figure 11 shows the reception screen updated when the Add button is clicked.

[0041] 9, when the workpieces W remaining in the container 15 cannot be held using the existing holding pattern, the control device 2 displays a reception screen (S221). By the processing of step S221, the reception screen 100 shown in Fig. 7 is displayed on the display device 7. Then, the control device 2 waits for the creation process of the next step until the create button 117 displayed on the reception screen 100 is clicked (S222; No).

[0042] When the Create button 117 is clicked (S222; Yes), the control device 2 creates a candidate retention pattern based on the retention pattern (S223). The control device 2 then creates a reception screen 100 with a schematic diagram corresponding to the candidate retention pattern inserted therein according to a predefined format, and updates the reception screen 100 displayed on the display device 7 (S224). Through the processing of step S224, the reception screen 100 shown in FIG. 7 is updated to the reception screen 100 shown in FIG. 10. As shown in FIG. 10, the third area 115 of the updated reception screen 100 displays multiple schematic diagrams 130, each representing a candidate retention pattern created in step S223. The schematic diagrams 130 display check boxes 131 and priority entry fields 132 in association with each other. The control device 2 then waits for the next step of the additional registration process until the Add button 119 displayed on the reception screen 100 is clicked (S225; No). The user can select a candidate for a retention pattern to be additionally registered as a retention pattern by clicking a check box 131 displayed on the reception screen 100. Furthermore, the user can set the priority order by inputting information into an input field 132.

[0043] Based on the clicking of the Add button 119 (S225; Yes), at least one hold pattern candidate selected by the user from the plurality of hold pattern candidates displayed on the reception screen 100 is additionally registered in the hold pattern management table in accordance with the priority set by the user (S226). By the processing of step S226, the reception screen 100 shown in Fig. 10 is updated to the reception screen 100 shown in Fig. 11. As shown in Fig. 11, a schematic diagram 120 displayed in the third area 115 and corresponding to the hold pattern candidate selected by the user is added to the second area 113 of the updated reception screen 100.

[0044] The priority set by the user may be defined as a priority among candidates for a holding pattern to be additionally registered, or may be defined as a priority for the entire system including existing holding patterns. The definition of the priority can be switched according to a user instruction. Furthermore, the priority of an existing holding pattern can be changed at any timing according to a user instruction. For example, when the priority is defined as a priority among candidates for a holding pattern to be additionally registered, a priority lower than the priority assigned to an existing holding pattern is assigned to the candidate for a holding pattern to be additionally registered, according to the priority set by the user.

[0045] The control device 2 according to the first embodiment has the following advantages. That is, the control device 2 according to the first embodiment can present the user with a reception screen 100 including candidate holding patterns to be added and registered as holding patterns. The user does not need to teach a holding pattern by directly inputting coordinates or operating on the simulation screen. Instead, the user can simply confirm the presented candidate holding patterns, select a candidate holding pattern to be added and registered as a holding pattern, and click the Add button 119 to add (teach) the selected candidate holding pattern as a holding pattern. Therefore, even a user who is unfamiliar with setting up bulk picking can easily add (teach) a holding pattern for the workpieces W.

[0046] The above-described additional registration of holding patterns can be performed when the workpieces W remaining in the container 15 cannot be held by the existing holding patterns. Therefore, the new holding patterns added by additionally registering holding patterns are only patterns that can hold the target workpieces W that cannot be held by the existing holding patterns. In this way, only holding patterns that can hold workpieces W that cannot be held by the existing holding patterns can be cumulatively added, which prevents the number of holding patterns from increasing unnecessarily and gradually reduces the time that the robot 10 is stopped because it cannot hold the workpieces W.

[0047] Furthermore, the candidate holding patterns presented to the user are determined by simulation to have a high probability of holding the target workpiece W without interfering with surrounding objects. However, even if a candidate holding pattern is determined by simulation to be capable of holding the target workpiece W, it may not actually be able to hold the target workpiece W due to the influence of the state of the robot 10, the workpiece W, and the surrounding environment. In the first embodiment, rather than automatically adding the created candidate holding patterns as holding patterns, only the candidate holding patterns finally confirmed by the user can be added. This makes it possible to add only candidate holding patterns that are highly likely to be able to hold a workpiece W that cannot be held by existing holding patterns. This contributes to preferably preventing a situation in which the number of holding patterns is unnecessarily increased.

[0048] In the first embodiment, the holding pattern candidate creation unit 30 creates holding pattern candidates based on the holding pattern, a 3D model of the container 15, a 3D model of the target workpiece W, and a 3D model of the robot hand 11. This allows the user to narrow down and present candidates to holding patterns that can hold the target workpiece W, so that no matter which holding pattern the user selects, the user can additionally register a holding pattern that has a high possibility of holding the target workpiece W. However, because it is ultimately the user who determines whether the pattern can be held, at the stage of presenting candidates to the user, the holding pattern candidates may be presented to the user regardless of whether the pattern can hold the target workpiece W.

[0049] Therefore, the holding pattern candidate creation unit 30 may create holding pattern candidates based on the holding pattern alone or on the holding pattern and a three-dimensional model of the target workpiece W. Specifically, the holding pattern candidate creation unit 30 can create holding pattern candidates by adding or subtracting a predetermined offset amount to the holding pattern. Furthermore, the holding pattern candidate creation unit 30 can extract multiple holding pattern candidates by searching within a predetermined range from the holding pattern H1, and create one holding pattern candidate that is closest to the center of gravity of the target workpiece W from the extracted multiple holding pattern candidates.

[0050] In the first embodiment, the reception screen is displayed when the workpieces W remaining in the container 15 cannot be held using the existing holding pattern. However, the timing at which the reception screen is displayed can be set as desired, and for example, the reception screen may be displayed continuously after bulk unloading has started.

[0051] In the first embodiment, the process of creating a candidate holding pattern is executed when a create button displayed on the reception screen is clicked, i.e., when a user operation is triggered. However, the process may be executed automatically when it is determined that the workpieces W remaining in the container 15 cannot be held using the existing holding pattern.

[0052] In the reception screen 100 shown in Fig. 10, the user must determine which workpiece W' each of the multiple candidate holding patterns is presented as being capable of holding. Therefore, as shown in Fig. 12, the correspondence between the workpiece W' and the candidate holding patterns is displayed, for example, in a speech bubble, so that the user can see which of the multiple candidate holding patterns each of the multiple candidate holding patterns is capable of holding. This further reduces the effort required for the user to add candidate holding patterns. The method of displaying the correspondence between the workpiece W' and the candidate holding patterns capable of holding the workpiece W' is not limited to speech bubbles; for example, existing identification methods such as color or hatching can be used.

[0053] The reception screen 100 includes an execute button 111 so that a simulation can be performed to confirm whether the workpiece W' remaining in the container 15 can be held in a newly added or existing holding pattern. When the execute button 111 is clicked, a simulation is performed based on the image data, and information is displayed regarding whether the workpiece W' remaining inside the container 15 can be held in multiple holding patterns, including the newly added holding pattern candidate. For example, as shown in FIG. 13 , when the execute button 111 is clicked, a correspondence between the workpiece W' and the holding pattern is displayed in a speech bubble so that the user can see which holding pattern can hold the workpiece W' remaining in the container 15. This allows the user to reconfirm whether the newly added holding pattern can hold the workpiece W' remaining in the container 15'. If the added holding pattern cannot hold any of the target workpieces W', the holding pattern is unnecessary, and the newly added holding pattern may be deleted. This prevents the unnecessary proliferation of holding patterns. The method of displaying the correspondence between the workpiece W' and the holding pattern capable of holding the workpiece W' is not limited to speech bubbles, and existing identification methods such as color and hatching can be used.

[0054] Second Embodiment A control device 2' according to a second embodiment is a device in which a function for adjusting candidates for holding patterns is added to the control device 2 according to the first embodiment. The hardware configuration of the control device 2' according to the second embodiment is similar to the hardware configuration of the control device 2 according to the first embodiment, and therefore a description thereof will be omitted (see FIG. 2).

[0055] The control device 2 according to the second embodiment will be described below with reference to Figures 14, 15, 16, and 17. Figure 14 is a functional block diagram of the control device 2 according to the second embodiment. As shown in Figure 14, the control device 2' according to the second embodiment is configured by adding an adjustment unit to the control device 2 according to the first embodiment (see Figure 3).

[0056] The reception unit 21 receives a user instruction regarding adjustment of a plurality of holding pattern candidates as a user operation on a reception screen. The adjustment unit 33 adjusts the holding pattern candidates in accordance with the adjustment instruction from the user received by the reception unit 21.

[0057] Fig. 15 shows an example of a reception screen presented to a user by the control device 2 according to the second embodiment. Fig. 16 is a diagram showing the reception screen when the cursor is aligned with the interference information in Fig. 15. Fig. 17 is a diagram showing the reception screen when the adjustment button is clicked in Fig. 15. Fig. 15 corresponds to Fig. 10 of the first embodiment.

[0058] As shown in FIG. 15 , in a reception screen 100′ presented to a user by the control device 2 according to the second embodiment, a schematic diagram 130 displays a check box 131, a priority input field 132, an adjustment button 133, and interference information 134 in association with each other. The interference information 134 is information regarding interference between the robot hand 11 and peripheral objects of the target workpiece W. For example, as shown in FIG. 15 , if the robot hand 11 interferes with a peripheral object, "interference occurs," and if there is no interference, "no interference" is displayed as the interference information 134. If there is no interference, the interference information 134 may be hidden. Note that peripheral objects include the container 15 and other workpieces W other than the target workpiece W.

[0059] 15, when the cursor is positioned on the interference information 134, a details screen 135 describing the details of the interference information is displayed as a pop-up, as shown in Fig. 16. Note that the details of the interference information may be displayed on the reception screen 100' in place of the interference information 134 "Interference exists."

[0060] When the Adjust button 133 in FIG. 15 is clicked, an adjustment screen 136 for accepting adjustments to the holding pattern candidates is popped up as shown in FIG. 17 . The adjustment screen 136 includes a virtual space in which a 3D model of the workpiece W, a 3D model of the container 15, and a 3D model of the robot hand 11 are arranged based on the holding pattern candidates. The user can adjust the holding pattern candidates by inputting movement and rotation operations on the 3D model of the robot hand 11, and can overwrite the holding pattern candidates with the adjusted holding pattern candidates by clicking the Save button 137. The user can additionally register the adjusted holding pattern candidates as holding patterns. The adjustment method is not limited to this, as long as the holding pattern candidates can be adjusted. For example, the coordinates of the holding position and the holding posture corresponding to the holding pattern candidates may be displayed so that these coordinates can be adjusted.

[0061] The control device 2 according to the second embodiment achieves the following advantages. Specifically, the control device 2 according to the first embodiment can present only candidate holding patterns that do not interfere with surrounding objects to the user. This allows the user to easily add and register a holding pattern capable of holding the target workpiece W without causing the robot hand 11 to interfere with surrounding objects simply by selecting one of the presented candidate holding patterns. However, in cases where surrounding objects are complexly positioned around the target workpiece W, it may not be possible to automatically generate candidate holding patterns capable of holding the target workpiece W. The control device 2 according to the second embodiment can display not only candidate holding patterns that do not interfere with surrounding objects but also candidate holding patterns that do interfere with surrounding objects as options for the user to select, and can then adjust the candidate holding patterns according to user instructions. This effectively avoids the situation described above where candidate holding patterns capable of holding the target workpiece W cannot be automatically generated.

[0062] The various data stored in the storage device 9 may be recorded on removable media and distributed to users, or may be distributed by being downloaded to the control device 2 via the network W.

[0063] The following supplementary notes are further disclosed regarding this embodiment and its modified examples. (Supplementary Note 1) A control device 2 that controls a robot to perform the operation of picking up bulk workpieces W includes a memory unit 24 that stores data on multiple holding patterns for holding the workpieces W, a selection unit 29 that selects one of the multiple holding patterns for the workpieces, a control unit 25 that controls the robot based on the selected holding pattern, a creation unit 30 that creates multiple holding pattern candidates based on the holding patterns, a reception unit 21 that receives a user instruction to add at least one of the created holding pattern candidates as a holding pattern, and a memory control unit 31 that stores the received holding pattern candidate in the memory unit 24 as a holding pattern. (Supplementary Note 2) The creation unit 30 of Supplementary Note 1 creates multiple holding pattern candidates in response to a user instruction. (Supplementary Note 3) The control device 2 described in Supplementary Note 1 further includes an acquisition unit 23 that acquires image data of the workpieces W, and a determination unit 28 that determines the feasibility of holding the workpieces W using the holding patterns added to the memory unit 24 based on the image data. (Supplementary Note 4) The creation unit 30 of Supplementary Note 3 creates multiple holding pattern candidates when the holding possibility for any of the multiple holding patterns is lower than a predetermined value. (Supplementary Note 5) In the control device 2 of Supplementary Note 1, the memory unit 24 further stores data on a three-dimensional model of the workpiece W, and the creation unit 30 creates multiple holding pattern candidates based on the holding pattern and the data on the three-dimensional model of the workpiece W. (Supplementary Note 6) The memory unit 24 of Supplementary Note 5 further stores data on a three-dimensional model of a hand for holding the workpiece W, and the creation unit 30 creates multiple holding pattern candidates based on the holding pattern, the data on the three-dimensional model of the workpiece W, and the data on the three-dimensional model of the hand. (Supplementary Note 7) In the control device 2 of Supplementary Note 1, the holding pattern includes at least a holding position at which the workpiece W is held, and the creation unit 30 creates holding pattern candidates within a predetermined range from the holding position of the holding pattern. (Supplementary Note 8) The reception unit 21 of Supplementary Note 7 receives a user designation of the predetermined range.(Supplementary Note 9) The control device 2 of Supplementary Note 1 further includes an acquisition unit 23 that acquires imaging data of the workpiece W, and a determination unit 28 that determines, for each of a plurality of holding pattern candidates, a possibility of interference of the workpiece W with a peripheral object based on the imaging data. (Supplementary Note 10) The control device 2 of Supplementary Note 9 further includes a display unit 22 that displays a reception screen for receiving a user instruction to add at least one of the created holding pattern candidates as a holding pattern, and only holding pattern candidates that have been determined to have a possibility of interference with a peripheral object lower than a predetermined value are displayed on the reception screen. (Supplementary Note 11) The control device 2 of Supplementary Note 9 further includes a display unit 22 that displays a reception screen for receiving a user instruction to add at least one of the created holding pattern candidates as a holding pattern, and among the holding pattern candidates displayed on the reception screen, holding pattern candidates that have been determined to have a possibility of interference with a peripheral object higher than a predetermined value are displayed together with information about the peripheral object that has a high probability of interference. (Supplementary Note 12) In the control device 2 of any one of Supplements 9 to 11, the peripheral objects include either a container 15 that stores the workpiece W or another workpiece W different from the workpiece W. (Supplementary Note 13) In the control device 2 of any one of Supplements 1 to 12, a priority is set for each of the multiple holding patterns, and the reception unit 21 receives at least one of the created holding pattern candidates along with the priority, and the memory control unit 31 adds the received holding pattern candidate to the multiple holding patterns according to the priority. (Supplementary Note 14) The reception unit 21 of the control device 2 of any one of Supplements 1 to 13 receives a user instruction regarding adjustment of the multiple holding pattern candidates. (Supplementary Note 15) A program is provided in a computer that is capable of controlling a robot to perform a pick-up operation for bulk workpieces W and that stores data regarding multiple holding patterns for holding workpieces W, and that implements the following: means for creating multiple holding pattern candidates based on the holding patterns; means for receiving a user instruction to add at least one of the created holding pattern candidates as a holding pattern; and means for storing the received holding pattern candidate as a holding pattern.

[0064] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the invention or the concept and spirit of the present invention derived from the content of the claims and their equivalents. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0065] 1...Robot system, 2...Control device, 5...Processor, 6...Operation device, 7...Display device, 8...Communication device, 9...Storage device, 10...Robot, 11...Robot hand, 13...3D vision camera, 15...Container, 21...Reception unit, 22...Display unit, 23...Acquisition unit, 24...Storage unit, 25...Robot control unit, 26...Work detection unit, 27...Target work determination unit, 28...Judgment unit, 29...Holding pattern selection unit, 30...Holding pattern candidate creation unit, 31...Storage control unit, 32...Screen creation unit, 33...Adjustment unit

Claims

1. A control device that controls a robot to perform a work of picking up bulk workpieces, a storage unit that stores data relating to a plurality of holding patterns for holding the workpiece by the end effector of the robot; a selection unit that selects one holding pattern from the plurality of holding patterns for the workpiece; a control unit that controls the robot based on the one holding pattern; a creation unit that creates a plurality of candidates for a retention pattern based on the retention pattern; a receiving unit that receives a user instruction to add at least one of the created candidates for the retention pattern as the retention pattern; a storage control unit that stores the received candidate for a holding pattern in the storage unit as the holding pattern; A control device comprising:

2. The control device according to claim 1 , wherein the creation unit creates the plurality of candidates for the holding pattern in response to an instruction from a user.

3. an acquisition unit that acquires imaging data of the workpiece; The control device according to claim 1 , further comprising: a determination unit that determines whether the workpiece can be held by the holding pattern added to the storage unit based on the image data.

4. The control device according to claim 3 , wherein the creating unit creates candidates for the plurality of holding patterns when the holdability of any of the plurality of holding patterns is lower than a predetermined value.

5. The storage unit further stores data of a three-dimensional model of the workpiece, The control device according to claim 1 , wherein the creation unit creates the plurality of holding pattern candidates based on the holding pattern and data of a three-dimensional model of the workpiece.

6. the storage unit further stores data of a three-dimensional model of a hand for holding the workpiece, The control device according to claim 5 , wherein the creation unit creates the plurality of candidates for the holding pattern based on the holding pattern, data of a three-dimensional model of the workpiece, and data of a three-dimensional model of the hand.

7. the holding pattern includes at least a holding position for holding the workpiece, The control device according to claim 1 , wherein the creation unit creates candidates for the holding pattern within a predetermined range from the holding position of the holding pattern.

8. The control device according to claim 7 , wherein the reception unit receives a user specification of the predetermined range.

9. an acquisition unit that acquires imaging data of the workpiece; The control device according to claim 1 , further comprising: a determination unit that determines, for each of the plurality of holding pattern candidates, a possibility of interference between the workpiece and a peripheral object based on the imaging data.

10. a display unit that displays a reception screen for receiving a user instruction to add at least one of the created candidates for the holding pattern as the holding pattern; The control device according to claim 9 , wherein only holding pattern candidates determined to have a probability of interference with the surrounding object lower than a predetermined value are displayed on the reception screen.

11. a display unit that displays a reception screen for receiving a user instruction to add at least one of the created candidates for the holding pattern as the holding pattern; The control device according to claim 9, wherein, among the candidate holding patterns displayed on the reception screen, a candidate holding pattern determined to have a likelihood of interference with the surrounding object higher than a predetermined value is displayed together with information about the surrounding object with a high likelihood of interference.

12. The control device according to claim 9 , wherein the surrounding object includes either a container that houses the workpiece or another workpiece different from the workpiece.

13. A priority order is set for each of the plurality of holding patterns, the receiving unit receives at least one of the created candidates for the retention pattern together with a priority; The control device according to claim 1 , wherein the storage control unit adds the accepted candidate holding patterns to the plurality of holding patterns in order of priority.

14. The control device according to claim 1 , wherein the reception unit receives a user instruction regarding adjustment of the plurality of holding pattern candidates.

15. a computer that can control a robot to perform a pick-up operation of a bulk piled workpiece, the computer having a storage device that stores data relating to a plurality of holding patterns for holding the workpiece; means for generating a plurality of candidates for a holding pattern based on the holding pattern; means for receiving a user instruction to add at least one of the created candidates for a retention pattern as the retention pattern; means for storing the received candidate for a retention pattern as the retention pattern in a storage device; A program to achieve this.