Retention parameter estimation device and retention parameter estimation method

Abstract

This holding parameter estimation device has an interface and a control unit. The interface obtains information pertaining to the end effector, holding target information and depth data. The control unit obtains an end effector model on the basis of said information. The control unit estimates the width of opening of a holding part in order to hold the holding target object, on the basis of the end effector model, the holding target information and the depth data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2021-128476, filed on August 4, 2021, the entire disclosure of which is incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to a retention parameter estimation device and a retention parameter estimation method. [Background technology]

[0003] BACKGROUND ART Conventionally, there is known a learning device that performs learning to appropriately determine the position and posture for grasping an object from an image on a computer (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-205929 Summary of the Invention

[0005] A retention parameter estimation device according to a first aspect comprises: an acquisition unit that acquires information about an end effector having a holding unit that holds a holding object with an arbitrary opening width, holding object information that indicates the holding object, and depth data about the holding object; The device is equipped with a control unit that acquires an end effector model indicating the area where the end effector may be present based on the information, and estimates the opening width of the holding unit to hold the object to be held based on the end effector model, the held object information, and the depth data.

[0006] The retention parameter estimation method according to the second aspect includes: Acquire information about an end effector having a holding portion that holds a holding object, holding object information indicating the holding object, and depth data about the holding object; obtaining an end effector model that indicates a region where the end effector may be present based on the information; An opening width of the holding portion for holding the object to be held is estimated based on the end effector model, the holding object information, and the depth data. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Figure 2] FIG. 10 is a side view showing an example in which an object to be held is held by an end effector. [Figure 3] 1 is a block diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a mask image representing the outline of a holding object on a plane on which the holding object is held. [Figure 5A] FIG. 10 is a diagram showing a model representing the finger opening width of an end effector model whose holding part has two fingers. [Figure 5B] FIG. 10 is a diagram showing a model representing the finger stroke of an end effector model having a holding part with two fingers. [Figure 5C] FIG. 10 is a diagram showing an entire model of an end effector model in which the holding part has two fingers, with the finger opening width and stroke combined. [Figure 6A] FIG. 10 is a diagram showing a model representing the finger opening width of an end effector model having a holding part with three fingers. [Figure 6B] FIG. 10 is a diagram showing a first model representing a finger stroke of an end effector model whose holding part has two fingers. [Figure 6C] FIG. 10 is a diagram showing a second model representing the finger stroke of an end effector model whose holding part has two fingers. [Figure 7] FIG. 2 is a diagram showing an example of an approach map. [Figure 8]FIG. 10 is a diagram illustrating a method for generating a second region of the approach map. [Figure 9] FIG. 10 is a diagram showing an approach map in the process of being created in which an object area and an obstacle area are arranged in a second area. [Figure 10] FIG. 8 is a diagram showing the approach map of FIG. 7 after dilation processing. [Figure 11] FIG. 10 is a diagram illustrating an example of a position at which an end effector model is projected onto an approach map. [Figure 12] FIG. 10 is a diagram showing positions where an end effector model can be projected onto an approach map and positions where it cannot be projected onto the approach map. [Figure 13] 10A and 10B are diagrams for explaining a method for estimating a possible existence area based on the projection of an end effector model onto an approach map. [Figure 14] FIG. 10 is a diagram showing the center positions of possible areas corresponding to each holder. [Figure 15] FIG. 1 is an external view of a specially shaped object to be held. [Figure 16] 16 is a diagram for explaining that the center position of a possible existence area based on the projection of an end effector model onto an approach map created for the protection object of FIG. 15 is deviated from the movable straight line. FIG. [Figure 17] 10A and 10B are diagrams illustrating a method for creating an opening width model based on an estimated opening width. [Figure 18] FIG. 2 is a diagram illustrating an example of a surrounding environment map. [Figure 19A] FIG. 10 is a diagram showing an object map representing the center of a holding object. [Figure 19B] FIG. 10 is a diagram showing an object map indicating the priority of holding positions designated by a user. [Figure 20] FIG. 10 is a diagram illustrating an example of a contact map. [Figure 21] 10 is a flowchart illustrating an example of a procedure for estimating a retention parameter. [Figure 22] 22 is a flowchart showing an example of a procedure for correcting and learning the holding position estimated by the holding parameter estimation method of FIG. 21. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of a retention parameter estimation device to which the present disclosure is applied will be described with reference to the drawings.

[0009] (Configuration example of robot control system 100) 1, 2, and 3, a robot control system 100 according to an embodiment of the present disclosure includes a robot 2, a camera 4, a robot control device 110, and a holding parameter estimation device 10. The robot 2 performs a task by holding a holding target 80 with an end effector 2B. The robot control device 110 controls the robot 2. In order for the robot 2 to hold the holding target 80, the holding parameter estimation device 10 estimates an opening width of the end effector 2B (described later) and a contact position when holding the object, as holding positions, and outputs the estimates to the robot control device 110.

[0010] In this embodiment, the robot 2 holds the object to be held 80 located on the work start table 6. That is, the robot control device 110 controls the robot 2 so that the object to be held 80 is held on the work start table 6. The robot 2 may move the object to be held 80 from the work start table 6 to the work target table 7. The object to be held 80 is also referred to as a work object. The robot 2 operates within the operating range 5.

[0011] <Robot 2> The robot 2 includes an arm 2A and an end effector 2B. The arm 2A may be configured as, for example, a six- or seven-axis vertical articulated robot. The arm 2A may be configured as a three- or four-axis horizontal articulated robot or a SCARA robot. The arm 2A may be configured as a two- or three-axis Cartesian robot. The arm 2A may be configured as a parallel link robot or the like. The number of axes configuring the arm 2A is not limited to those illustrated. In other words, the robot 2 has an arm 2A connected by multiple joints, and operates by driving the joints.

[0012] The end effector 2B may include, for example, a gripper configured to be able to hold the object 80 with any opening width. The gripper has at least one holding portion. Each holding portion may be configured to be movable along a predetermined direction.

[0013] The holding part may come into contact with the holding object 80 when holding the holding object 80. The holding part may have one or more joints. The holding part may be, for example, two or more fingers that pinch and hold the holding object 80. The fingers may be formed of members that are movable opposite to each other. Alternatively, the holding part may be at least one suction part that holds the holding object 80 by suction, or a scooping hand configured to be able to scoop up the holding object 80. The end effector 2B is not limited to these examples and may be configured to be able to perform various other operations. In the configuration illustrated in FIG. 1, the end effector 2B includes a gripper.

[0014] The robot 2 can control the position of the end effector 2B by operating the arm 2A. The end effector 2B may have an axis that serves as a reference for the direction in which it acts on the held object 80. If the end effector 2B has an axis, the robot 2 can control the direction of the axis of the end effector 2B by operating the arm 2A. The robot 2 controls the start and end of the operation of the end effector 2B acting on the held object 80. The robot 2 can move or process the held object 80 by controlling the operation of the end effector 2B while controlling the position of the end effector 2B or the direction of the axis of the end effector 2B. In the configuration illustrated in FIG. 1 , the robot 2 has the end effector 2B hold the held object 80 at the work start table 6 and moves the end effector 2B to the work target table 7. The robot 2 has the end effector 2B release the held object 80 at the work target table 7. In this way, the robot 2 can move the object to be held 80 from the work start table 6 to the work target table 7.

[0015] <Camera 4> In the exemplary configuration shown in FIG. 1, the robot control system 100 includes a camera 4 attached to the end effector 2B of the robot 2. The camera 4 captures an image of the holding object 80. For example, the camera 4 may capture an image of the holding object 80 from the direction in which the end effector 2B holds the holding object 80. The captured image of the holding object 80 is also referred to as a held object image. The camera 4 also includes a depth sensor and is configured to acquire depth data of the holding object 80. The depth data is data related to distances in different directions within the angle of view of the depth sensor. More specifically, the depth data can be considered information related to the distance from the camera 4 to a measurement point. The image captured by the camera 4 may include monochrome brightness information or brightness information for each color expressed by RGB (Red, Green, Blue), etc. The number of cameras 4 is not limited to one, and may be two or more. The camera 4 may also capture images of other objects located within a predetermined range from the holding object 80 as obstacles and acquire depth data of the obstacles. The camera 4 is not limited to being attached to the end effector 2B, but may be provided at any position where it can capture an image of the held object 80. In a configuration where the camera 4 is attached to a structure other than the end effector 2B, the above-mentioned image of the held object may be synthesized based on an image captured by the camera 4 attached to that structure. The image of the held object may be synthesized by image conversion based on the relative position and orientation of the end effector 2B with respect to the attachment position and orientation of the camera 4. Alternatively, the image of the held object may be generated from CAD and drawing data.

[0016] <Retention parameter estimation device 10> As shown in FIG. 3, the retention parameter estimation device 10 includes a control unit 12 and an interface (acquisition unit) 14.

[0017] The control unit 12 may include at least one processor to provide control and processing capabilities for executing various functions. The processor may execute programs that implement the various functions of the control unit 12. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be implemented as multiple integrated circuits and discrete circuits that are communicatively connected. The processor may also be implemented based on various other known technologies.

[0018] The control unit 12 may include a memory unit. The memory unit may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The memory unit stores various information. The memory unit stores programs to be executed by the control unit 12, etc. The memory unit may be configured as a non-transitory readable medium. The memory unit may function as a work memory for the control unit 12. At least a part of the memory unit may be configured as a separate entity from the control unit 12.

[0019] Based on information or data acquired by the interface 14, the control unit 12 estimates the opening width when displacing a portion of the end effector to allow the robot 2 to hold the object to be held 80. The opening width is the position of the portion of the holding unit that contacts the object to be held 80 in a predetermined direction relative to a reference position. The reference position is the closedest position in a predetermined direction to which the robot can move. In a configuration having at least one holding unit, the closedest position is, for example, the displaceable end in the predetermined direction that is capable of adsorbing the object to be held. In a configuration having multiple holding units, the closedest position is, for example, the displaceable end in the direction that approaches other holding units.

[0020] The estimation of the opening width is based on the end effector model, holding object information, and depth data, which will be described later. The holding object information is information indicating the position of the holding object when the camera captures the image, such as a holding object image. The holding object image in the following description is a concrete description of the holding object information and may include information other than an image. The control unit 12 may further estimate a holding position at which the robot 2 should hold the holding object 80, based on information or data acquired by the interface 14.

[0021] The interface 14 acquires information or data relating to the holding object 80, etc. from an external device. The interface 14 may be configured to include an input device that accepts input of information, data, etc. from a user. The input device may be configured to include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include an audio input device such as a microphone.

[0022] The interface 14, for example, acquires a holding target image obtained by capturing the holding target 80 from the camera 4 and depth data associated with the holding target image. The interface 14, for example, acquires information about the end effector 2B from the robot 2 or an input device.

[0023] The interface 14 may further output information or data to an external device. The interface 14 may output the opening width of the end effector 2B estimated by the control unit 12. The interface 14 may output the holding position at which the object to be held 80 is to be held, estimated by the control unit 12.

[0024] The interface 14 may output information or data to the user so that the user can recognize it. The interface 14 may be configured to include an output device that outputs information or data to the user. When executing control of the robot based on the estimated opening width, for example, the interface 14 may present the estimation result to the user using the output device and accept the user's instruction regarding whether or not to execute the control. The user's instruction may be acquired using the above-mentioned input device. The retention parameter estimation device 10 may output the estimated opening width, etc. to the robot control device 110 without requesting a user's instruction based on the estimation result of the estimated opening width.

[0025] The output device may include, for example, a display device that outputs visual information such as images, characters, or graphics. The display device may include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display or an inorganic EL display, a PDP (Plasma Display Panel), or the like. The display device is not limited to these displays and may include displays of various other types. The display device may include a light-emitting device such as an LED (Light Emission Diode) or an LD (Laser Diode). The display device may include various other devices. The output device may include, for example, an audio output device such as a speaker that outputs audio information such as sound. The output device is not limited to these examples and may include various other devices.

[0026] The interface 14 may include a communication device configured to be capable of wired or wireless communication. The communication device may be configured to be capable of communication using a communication method based on various communication standards. The communication device may be configured using known communication technology.

[0027] <Robot control device 110> The robot control device 110 may acquire information specifying the opening width from the holding parameter estimation device 10. The robot control device 110 may control the robot 2 to open the end effector 2B to the estimated opening width when a portion of the end effector 2B is displaced to an approach area described below. The robot control device 110 may further acquire information specifying a holding position from the holding parameter estimation device 10. The robot control device 110 may control the robot 2 to hold the holding target 80 at the estimated holding position.

[0028] The robot controller 110 may be configured to include at least one processor to provide control and processing power for executing various functions. Each component of the robot controller 110 may be configured to include at least one processor. A plurality of components of the robot controller 110 may be implemented by a single processor. The entire robot controller 110 may be implemented by a single processor. The processor may execute programs that implement various functions of the robot controller 110. The processor may be configured the same as or similar to the processor used in the retention parameter estimation device 10.

[0029] The robot control device 110 may include a memory unit. The memory unit may be configured in the same manner as or similar to the memory unit used in the retention parameter estimation device 10.

[0030] The robot control device 110 may include the retention parameter estimation device 10. The robot control device 110 and the retention parameter estimation device 10 may be configured as separate entities.

[0031] (Operation example of robot control system 100) The robot control system 100 controls the robot 2 using the robot control device 110 to cause the robot 2 to perform a task. In this embodiment, the task to be performed by the robot 2 includes an operation for holding the holding object 80. The task to be performed by the robot 2 may further include an operation for holding the holding object 80. In the robot control system 100, the holding parameter estimation device 10 estimates an opening width when the end effector 2B is displaced toward the holding object 80. The robot control device 110 may control the robot 2 to open the end effector 2B to the estimated opening width. In the robot control system 100, the holding parameter estimation device 10 may further estimate a holding position of the holding object 80 by the robot 2. The robot control device 110 may control the robot 2 to hold the holding object 80 at the holding position.

[0032] In a configuration in which the end effector 2B has two or more fingers as a holding portion, the control unit 12 may estimate, as the holding position, a combination of positions where the holding portion comes into contact with the holding object 80 when the holding portion grasps the holding object 80. In a configuration in which the end effector 2B has an adsorption portion as a holding portion, the control unit 12 may estimate, as the holding position, a position where the adsorption portion comes into contact with the holding object 80 when the end effector 2B adsorbs the holding object 80.

[0033] The holding parameter estimation device 10 acquires, via the interface 14, a holding target image of the holding target 80 captured by the camera 4 and depth data of the holding target 80. The control unit 12 recognizes the outer shape and position of the holding target 80 based on the holding target image and the depth data. As illustrated in FIG. 4 , the control unit 12 generates a mask image 20 representing the recognition result of the holding target 80 when viewed from the camera 4 attached to the end effector 2B. The mask image 20 includes a window 22 representing the area where the holding target 80 exists as viewed from the camera 4, and a mask 24 representing the other area. In FIG. 4 , the window 22 is represented as a white area. In FIG. 4 , the mask 24 is represented as a hatched area with diagonal lines slanting upward to the right. Note that, for convenience of illustration, the mask 24 portion of the mask image 20 is represented as a hatched area, but in a real embodiment, it may be represented as a black area. Alternatively, in the mask image 20, the coloring of the window 22 and the mask 24 may be reversed. Alternatively, in the mask image 20, the coloring and hatching of the window 22 and the mask 24 may be reversed.

[0034] The control unit 12 acquires information about the end effector 2B via the interface 14. The information about the end effector 2B includes, for example, information specifying the maximum distance between the gripping holders, the thickness of the holders, and the width of the holders. The thickness of the holders is the length in the opening / closing direction of the holders. The width of the holders is the length in a direction perpendicular to the opening / closing direction of the holders. The control unit 12 may generate an end effector model 30 based on the information about the end effector 2B. The end effector model 30 indicates an area in which the end effector 2B may exist. The control unit 12 may acquire the end effector model 30 via the interface 14 as information about the end effector 2B.

[0035] As shown in FIG. 5A, an end effector model 30 having a two-fingered gripper includes a gripper model that identifies a gripper position 32 representing the range in which the grippers of the gripper are positioned with a predetermined distance between them and a non-operating range 38 representing the range other than the gripper position 32. In the case of an end effector model 30 having a three-fingered gripper, the end effector model 30 may include a gripper model that identifies a gripper position 32 representing the range in which the grippers of the gripper are positioned with a predetermined distance between them and a non-operating range 38 representing the range other than the gripper position 32, as shown in FIG. 6A. In other words, the gripper model represents the opening width of the gripper's grippers. The gripper position 32 may represent the range in which the grippers are positioned with the maximum distance between them. In this case, the gripper model represents the maximum opening width of the grippers. The gripper models shown in FIGS. 5A and 6A represent the maximum opening width of the grippers. The gripper position 32 is represented as a white area. The non-operating range 38 is represented as an area hatched with diagonal lines slanting upward to the right. Note that the non-operating range 38 is represented as a hatched area for convenience of illustration in the drawings, but in an actual embodiment, it may be represented as a black area. Alternatively, the coloring of the holder positions 32 and the non-operating range 38 may be reversed. Alternatively, the coloring and hatching of the holder positions 32 and the non-operating range 38 may be reversed.

[0036] An end effector model 30 having a gripper with two fingers includes a stroke model specifying a stroke range 34 representing the range in which the gripper's gripper operates, as shown in FIG. 5B . In the case of an end effector model 30 having a gripper with three fingers, the end effector model 30 may include a stroke model specifying a stroke range 34 representing the range in which the gripper's gripper operates, as shown in FIG. 6B or 6C . The stroke range 34 is represented as a white area. The non-operating range 38 is represented as a hatched area with diagonal lines slanting upward to the right. Although the non-operating range 38 is represented as a hatched area for convenience of illustration, it may be represented as a black area in a real embodiment. Alternatively, the coloring of the stroke range 34 and the non-operating range 38 may be reversed. Alternatively, the coloring and hatching of the stroke range 34 and the non-operating range 38 may be reversed.

[0037] As shown in FIG. 5C, the end effector model 30 includes an overall model that combines the gripper model of FIG. 5A and the stroke model of FIG. 5B. The overall model specifies a gripper operating range 36. The gripper operating range 36 includes a gripper position 32. Note that in FIG. 5C, the gripper portion of the gripper operating range 36 is shown as being distinguished by a dashed line, but this distinction may not be necessary in an actual embodiment. In addition, in this embodiment, the end effector model 30 represents a range in which the grippers are positioned with the maximum spacing (maximum opening width), but this is not limited to this. The end effector model 30 may represent a range in which the grippers of the gripper are positioned with any spacing (predetermined spacing). For example, the end effector model 30 may represent the spacing of the gripper grippers according to the size of the object to be grasped by the end effector 2B.

[0038] The control unit 12 may generate only the overall model as the end effector model 30. In this case, the holder position 32 may be identified by associating information identifying the holder position 32 with the overall model. The information identifying the holder position 32 may include numerical values ​​representing characteristic points of the holder.

[0039] In a configuration in which the holding part has an adsorption part, the end effector model 30 is configured as a model that defines the range in which the adsorption part interferes with other objects when adsorbing the object 80 to be held.

[0040] The control unit 12 sets the height at which the holding object 80 is held based on the depth data of the holding object 80. Specifically, as illustrated in FIG. 2 , the control unit 12 sets the height from the work start platform 6 as a position at which the holding object 80 placed on the work start platform 6 is held. In FIG. 2 , the position at which the holding object 80 is clamped and held by the holding portion of the end effector 2B is represented as a holding point 82. The height of the holding point 82 from the work start platform 6 is represented as H. The control unit 12 sets H as the height at which the holding object 80 is held. Based on the depth data of the holding object 80, the control unit 12 sets the height at which the holding object 80 is held to a value smaller than the distance from the work start platform 6 to the highest point of the holding object 80. The control unit 12 may also set the height at which the holding object 80 is held to a value approximately half the height of the holding object 80.

[0041] The control unit 12 may generate the mask image 20 based on the height at which the holding object 80 is held and depth data of the holding object 80. Specifically, the control unit 12 may generate the mask image 20 in which the cross-sectional shape of the holding object 80 in a plane at the height at which the holding object 80 is held is defined as the window 22.

[0042] The control unit 12 may create an approach map for specifying the opening width when a part of the end effector 2B is displaced in order to hold the holding object 80. The control unit 12 may create the approach map based on the holding object image and depth data.

[0043] As shown in FIG. 7 , the approach map 90 may show at least an approach region 91. The approach map 90 may further show an object region 92, a non-approach region 93, and a first region 94. The approach region 91 is a region where the end effector 2B can be opened at a height for holding the holding object 80 without interfering with objects other than the holding object 80. The object region 92 is a region where the holding object 80 exists. The non-approach region 93 is a region outside the second region described below, and is the entire region of the approach map 90 excluding the approach region 91, the object region 92, and the first region 94. The first region 94 is a region facing away from the outer edge facing the holding object 80 in a region where an object other than the holding object 80 exists, in other words, an obstacle exists. The approach region 91 is shown as a white region. The object region 92, the non-approach region 93, and the first region 94 are represented as regions hatched with diagonal lines slanting upward to the right. The object region 92, the non-approach region 93, and the first region 94 are represented as hatched regions for the convenience of illustrating the drawings, but in an actual embodiment, they may be represented as black regions. Alternatively, the coloring of the approach region 91, the object region 92, the non-approach region 93, and the first region 94 may be reversed. Alternatively, the coloring and hatching of the approach region 91, the object region 92, the non-approach region 93, and the first region 94 may be reversed.

[0044] To create the approach map 90, the control unit 12 may generate a second region based on the mask image 20. The second region is a region in which the end effector 2B can be opened around the holding object 80 without interfering with the holding object 80, focusing only on the holding object 80. In other words, the second region may be a region from the outer edge of the holding object 80 that has the maximum opening width of the holding portion of the end effector 2B. The control unit 12 may generate the second region by convolving the end effector model 30 with the mask image 20. More specifically, the control unit 12 may move the holding unit's operating range 36, specified by the end effector model 30, so that at least a portion of the holding unit's operating range 36 overlaps with the window 22 included in the mask image 20. When moving the holding unit's operating range 36, the rectangle representing the holding unit's operating range 36 may be rotated at various angles so that at least a portion of the rectangle overlaps with the upper left corner of the window 22, for example. The control unit 12 may generate, as the second area, an area through which the movement range 36 of the holder passes when the movement range 36 of the holder is moved. In Fig. 8, the trajectory of the point farthest from the window 22 when the movement range 36 of the holder moves is represented as the boundary 95 of the second area. The boundary 95 is represented by a dashed line.

[0045] 9, the control unit 12 generates an object region 92 in the generated second region 96 based on depth data of the holding object 80. The control unit 12 generates a range in which the holding object 80 exists at a position higher than the height at which the holding object 80 is held as the object region 92. The control unit 12 also places an obstacle region 97 in the generated second region 96 based on depth data of objects other than the holding object 80. The control unit 12 generates a range in which objects other than the holding object 80 exist at a position higher than the height at which the holding object 80 is held as the obstacle region 97.

[0046] The control unit 12 generates a first region 94 based on the object region 92 and the obstacle region 97. The first region 94 is a region that includes the obstacle region 97 and extends outside the obstacle region 97, and as described above, is a region that faces away from the holding object 80 from the outer edge of the obstacle region 97 that faces the holding object 80. The control unit 12, for example, calculates a straight line that passes through the center C92 of the object region 92 and overlaps the obstacle region 97. The control unit 12 rotates the straight line around the center C92 as an axis to generate, as the first region 94, a trajectory along which a line segment of the straight line that is outside the obstacle region 97 passes. Alternatively, the control unit 12 draws two straight lines SL that pass through the center C92 of the object region 92 and intersect the outer edge of the obstacle region 97 at one point. As shown in FIGS. 7 and 9, the control unit 12 generates a first region 94 that is an area surrounded by the two straight lines SL, the outer edge of the second region 96, and the outer edge of the obstacle region 97 on the object region 92 side.

[0047] The control unit 12 generates the approach region 91 by excluding the object region 92 and the first region 94 from the second region 96. As shown in FIG. 10 , the control unit 12 may generate the approach region 91 by excluding at least one of a region 92ex obtained by expanding the object region 92 and a region 94ex obtained by expanding the first region 94 from the second region 96. The object region 92 and the first region 94 excluded from the second region 96 may be expanded by at least half the width of the holding portion in the width direction of the holding portion and by at least half the thickness of the holding portion in the thickness direction of the holding portion. In the following description, the object region 92 or the expanded region 92ex may also be referred to as the “object region 92, 92ex.” Furthermore, the first region 94 or the expanded region 94ex may also be referred to as the “first region 94, 94ex.”

[0048] The control unit 12 estimates the opening width at an arbitrary point within the range of the window 22 in the mask image 20. The arbitrary point is, for example, a reference position of the end effector 2B when the end effector 2B is caused to hold the holding object 80. The arbitrary point selected within the range of the window 22 corresponds to an approach position 70 included in the object region 92, 92ex in the approach map 90, as shown in FIG. 11. That is, the control unit 12 sets the arbitrary point within the range of the window 22 in the mask image 20 as the approach position 70. The control unit 12 projects the end effector model 30 with its center aligned with the approach position 70. The projected end effector model 30 is represented as projection models 72a and 72b in FIG. 11. The projection model 72a corresponds to a model obtained by rotating the end effector model 30 along the short side direction of the object region 92, 92ex. The projection model 72b corresponds to a model obtained by rotating the end effector model 30 by 45 degrees clockwise from the direction of the short side of the object regions 92, 92ex.

[0049] The control unit 12 determines that the end effector model 30 cannot be projected when the holder position 32 included in the end effector model 30 overlaps the object region 92, 92ex. As shown in FIG. 12 , the projection position 74a of the holder included in the projection model 72a does not overlap the object region 92, 92ex. Therefore, the control unit 12 can project the projection model 72a onto the approach map 90. On the other hand, the projection position 74c of the holder included in the projection model 72c overlaps the object region 92, 92ex. Therefore, the control unit 12 cannot project the projection model 72c onto the approach map 90. By not projecting the end effector model 30 when the holder position 32 overlaps the object region 92, 92ex, the approach position 70 at which the end effector 2B collides with the held object 80 while approaching the held object 80 is determined. By determining the approach position 70 where the end effector 2B impacts the object 80, positions that should be excluded as arbitrary points to select for estimation are determined.

[0050] Furthermore, the control unit 12 determines that the end effector model 30 cannot be projected if it can be determined that the distance between the object region 92 and the first region 94 is shorter than the thickness of the holding portion. The distance can be determined to be shorter than the thickness of the holding portion when, in the approach map 90 in which either the object region 92 or the first region 94 is expanded, the distance is shorter than half the thickness of the holding portion. Alternatively, the distance can be determined to be shorter than the thickness of the holding portion when, in the approach map 90 in which both the object region 92 and the first region 94 are expanded, the expanded regions 92ex and 94ex overlap even slightly. With this configuration, the approach position 70 at which the end effector 2B collides with an obstacle is determined when a portion of the end effector 2B is displaced toward the holding object 80 in order for the end effector 2B to hold the holding object 80. By determining the approach position 70 where the end effector 2B collides with the obstacle, positions that should be excluded as arbitrary points to be selected for estimating the opening width are determined.

[0051] In a configuration in which the holding part position 32 is not specified in the end effector model 30, the control unit 12 determines whether the end effector model 30 can be projected based on the characteristic points of the holding part associated with the end effector model 30.

[0052] The position of the end effector 2B corresponding to the position and rotation angle of the end effector model 30 that can be projected onto the approach map 90 can also be said to be a position that fits the approach map 90. It can also be said that the control unit 12 estimates the opening width for each position that fits the approach map 90.

[0053] When the control unit 12 is able to project the end effector model 30 onto the approach map 90, it may estimate the opening width based on an approach region 91 in the approach map 90 and a region in the end effector model 30 where the end effector 2B can be present. As shown in Fig. 13 , the control unit 12 estimates a possible presence region 99 for each holder in order to estimate the opening width. The possible presence region 99 is the region where the approach region 91 and the operating range 36 of the holder overlap when the end effector model 30 is overlaid on the approach map 90.

[0054] The control unit 12 may estimate any point within the possible presence area 99 of the holding unit as the position of the holding unit corresponding to the opening width. The control unit 12 estimates the position of each holding unit in each possible presence area 99 corresponding to each holding unit.

[0055] The control unit 12 may estimate the position of the holding unit corresponding to the opening width, for example, by the following method. As shown in FIG. 14 , the control unit 12 calculates a first center position CP1, which is the center position of the possible presence area 99 corresponding to each holding unit. The control unit 12 determines whether or not the holding unit interferes with the holding target 80 and objects other than the holding target 80 when the holding unit is displaced along a predetermined direction of each holding unit from the first center position CP1 as a starting point. The control unit 12 displaces the holding unit from the first center position CP1 as a starting point until it reaches the edge of the corresponding possible presence area 99. In other words, the control unit 12 estimates, as the position of the holding unit corresponding to the opening width in the possible presence area 99, any position that does not interfere with the holding target 80 and objects other than the holding target 80, within the area through which the holding unit passes until it reaches the edge of the possible presence area 99. In other words, the control unit 12 estimates an arbitrary position within the area where the area through which the holding unit passes before reaching the edge of the possible presence area 99 overlaps with the approach area 91 as the position of the holding unit corresponding to the opening width in the possible presence area 99. More specifically, the control unit 12 displaces the first center position CP1 along a predetermined direction. Furthermore, the control unit 12 estimates an arbitrary point displaced from the first center position CP1 that can be located within the approach area 91 as the position of the holding unit.

[0056] In addition, when the end effector 2B has multiple holding parts that open and close to the same width, the position of each holding part corresponding to the opening width is estimated to be a position where the opening width is the same for all holding parts.

[0057] As shown in FIG. 15 , an object having a main body 81 and a bracket member 83 in cross section at an arbitrary height may be the holding target 80. For a holding target 80 having such a shape, depending on the approach position 70, as shown in FIG. 16 , the first center position CP1 of the possible presence area 99 corresponding to one holding unit may deviate from the center in the width direction of the holding unit. In other words, the first center position CP1 may deviate from a movable straight line AL that passes through the center position (approach position 70) of the area where the end effector 2B can be present and is parallel to the opening and closing direction of the holding unit. When the first center position CP1 deviates from the movable straight line AL, as in the above example, the control unit 12 may regard the intersection of the movable straight line AL and a perpendicular line drawn from the first center position CP1 to the movable straight line AL as the first center position CP1, and estimate the position of the holding unit corresponding to the opening width, as described above.

[0058] The control unit 12 may first displace the holding unit from the first center position CP1 toward a region in the end effector model 30 where the end effector 2B can exist, in other words, toward the center position of the holding unit's operating range 36. When displacing the holding unit toward the center position and the approach region 91 does not exist, the control unit 12 may search for the approach region 91 outside the first center position CP1 in the possible existence region 99.

[0059] For example, the control unit 12 may select an arbitrary position where the point first reaches the approach region 91 while displacing the point along a predetermined direction starting from the first center position CP1, and estimate the position as the position of the holding unit. For example, as shown in Fig. 14, if the possible presence region 99 corresponding to any single holding unit has the same width as the holding unit over the entire area in the direction along the movable straight line AL, the first center position CP1 may be estimated as the position of the holding unit.

[0060] The control unit 12 moves the approach position 70 so as to scan within the range of the window 22 of the mask image 20, and rotates the end effector model 30 at each position to project it onto the approach map 90. The control unit 12 extracts a combination of the approach position 70, the rotation angle of the end effector model 30, and the opening width of the end effector 2B that allows the end effector model 30 to be projected onto the approach map 90.

[0061] The control unit 12 may create an opening width model based on the estimated opening width. The opening width model specifies an area where the end effector 2B may be located within the range of the opening width estimated as the position of the holding part. As shown in FIG. 17 , the opening width model 31 is generated by drawing, for each holding part, an area having the width of the holding part, with a length from the center of the holding part's operating range 36 to the holding part position WP corresponding to the opening width.

[0062] The control unit 12 may estimate a holding position where the end effector 2B comes into contact with the holding object 80 based on the opening width model 31 and a rule map described later. The control unit 12 may generate a rule map to estimate the holding position. The rule map may be generated for each height at which the holding object 80 is held. The rule map may specify rules based on the display format of an image expressed two-dimensionally. The control unit 12 may generate the rule map based on at least one of the holding object image and depth data. The control unit 12 may acquire the rule map from an external device via the interface 14.

[0063] The rule map may include a map that defines the position of the holding object 80 that the end effector 2B should use for holding, in other words, the position that the end effector 2B should contact, i.e., the holding position. The rule map may include a map that is generated based on the height at which the holding object 80 is held. The rule map may be classified into, for example, a surrounding environment map 40 (see FIG. 18), an object map 50 (see FIGS. 19A and 19B), or a contact map 60 (see FIG. 20). The control unit 12 may acquire the rule map based on at least one of the shape data of the holding object 80 and the depth data associated with the holding object image.

[0064] 18 , the surrounding environment map 40 may identify an approach region 41, an object region 42, a non-approach region 43, and a first region 44. The surrounding environment map 40 may be the same as the approach map 90 that has not been subjected to expansion processing. Therefore, the approach region 41, the object region 42, the non-approach region 43, and the first region 44 in the surrounding environment map 40 may be the same as the approach region 91, the object region 92, the non-approach region 93, and the first region 94 in the approach map 90, respectively. The surrounding environment map 40 may be generated by the same method as the approach map 90.

[0065] The control unit 12 may perform a blurring process on the generated surrounding environment map 40 to blur the boundaries. Each region included in the surrounding environment map 40 generated by the above-described procedure can be distinguished by the numerical values ​​at each coordinate in the map. For example, the control unit 12 may set the numerical value of a coordinate included in the approach region 41 to 1, indicating that the point identified by that coordinate is included in the operating range of the end effector 2B. On the other hand, the control unit 12 may set the numerical value of a coordinate included in the object region 42, the non-approach region 43, and the first region 44 to 0 (zero), indicating that the point identified by that coordinate is not included in the operating range of the end effector 2B. The control unit 12 sets the numerical value of the coordinate that identifies a point within a predetermined range from the boundary between the region where 1 is set and the region where 0 is set to a value greater than 0 but less than 1, for example, 0.5. The control unit 12 performs the process of blurring the boundaries of the regions in this way as a blurring process.

[0066] Furthermore, the surrounding environment map 40 generated by the above-described procedure can be distinguished by the color of each coordinate in the map. For example, the control unit 12 may represent points included in the approach area 41 in white, and points included in other areas in black. The control unit 12 may represent the color of points within a predetermined range from the boundary between the area represented by white and the area represented by black in grayscale. The control unit 12 may perform a process to blur the boundary between areas in this way as a blurring process. Representing the color of each area in black, white, and gray is equivalent to representing the numerical value set for each area as a brightness value.

[0067] By performing the blurring process, the control unit 12 can reduce the possibility of the end effector 2B unexpectedly colliding with an object due to an error in the operating range of the end effector 2B or an error in the contour of the held object 80 or an obstacle. In other words, the control unit 12 can estimate the position of the end effector 2B relative to the held object 80 by taking various margins into consideration through the blurring process. The blurring process may be performed on the peripheral portion of each area included in the generated surrounding environment map 40. Each area is enlarged by the blurring process.

[0068] The object map 50 represents information that is referenced to determine at what position on the holding object 80 the worker should hold the holding object 80. The object map 50 represents, for example, information such as the shape, material, or density distribution of the holding object 80.

[0069] For example, it is assumed that the holding object 80 has a proper holding position according to the rule as it approaches the center of the holding object 80. The object map 50 for the assumed holding object 80 may be expressed in a grayscale, as shown in FIG. 19A , in a cross section 52 obtained by cutting the holding object 80 along a plane at the height at which the holding object 80 is held, in which the closer to the center of the holding object 80, the more white the color, corresponding to a proper holding position according to the rule, and the farther from the center, the more black the color, corresponding to a holding position that is inappropriate according to the rule. Furthermore, as shown in FIG. 19B , the object map 50 may be expressed in a grayscale that specifies a rule arbitrarily set by the user as a rule for estimating the holding position of the holding object 80. When creating an object map 50 in which the proper holding position is obtained as it approaches the center of the holding object 80, as shown in FIG. 19B , the closer to the center along the horizontal direction of the cross section 52 of the holding object 80, the more white the color, corresponding to a proper holding position, and the farther from the center, the more black the color, corresponding to an inappropriate holding position. Moreover, the object map 50 in FIG. 19B is displayed in the same color along the height direction of the cross section 52.

[0070] 19A and 19B, the solid black lines surrounding the cross sections 52 of the object map 50 are lines that simply represent the outline of the cross sections 52 and do not indicate rules. They do not indicate that the coordinates where the solid black lines are drawn are incorrect holding positions.

[0071] The object map 50 may be configured such that, when assuming a position for holding the object 80 and evaluating the appropriateness of holding the object at the assumed position, an appropriateness value indicating the evaluation of appropriateness increases when the object is held near an area represented by a color close to white. The object map 50 may also be represented by associating a numerical value with the color of each coordinate, as in the surrounding environment map 40. For example, the control unit 12 may set 1 to coordinates represented by white and 0 to coordinates represented by black.

[0072] The control unit 12 may generate the object map 50 in which a color or a numerical value is set for each coordinate to specify various rules, without being limited to the above example. For example, the control unit 12 may generate the object map 50 in which a color or a numerical value is set for each coordinate according to the distance from the center of gravity of the holding target 80. For example, the control unit 12 may generate the object map 50 in which a color or a numerical value is set for each coordinate to specify a holding position that should be avoided or a holding position that is prohibited.

[0073] The control unit 12 may generate one object map 50 that specifies multiple rules by combining object maps 50 that specify one rule. When combining multiple object maps 50, the control unit 12 may set a weighting coefficient for each object map 50. For example, if the center of gravity position of the holding target 80 is important, the control unit 12 may set a large weighting coefficient for the object map 50 that specifies the center of gravity position.

[0074] The object map 50 may be defined based on the characteristics of the holding object 80 itself. The object map 50 may be defined based on any of the shape, material, texture, weight, or friction coefficient of the holding object 80. The object map 50 may be based on rules arbitrarily set by a user regarding the holding position of the holding object 80. For example, parts of the holding object 80 that should not be used as holding positions for various reasons, such as parts that are easily damaged or deformed by contact, parts with grease or the like, or parts that are slippery and unsuitable for holding, may be defined as rules in the object map 50. Similarly, parts that are less likely to be damaged or deformed, parts without grease or the like, parts that are not slippery, and other parts that should be used as holding positions (parts that are considered easy to hold) based on empirical rules may be defined as rules in the object map 50. Representing the object map 50 as a two-dimensional image makes it easier to understand the rules defined for the holding position of the holding object 80. The object map 50 may be generated for each type of holding object 80.

[0075] The contact map 60 represents rules determined based on the relationship between the fingers of the end effector 2B and the surface condition of the holding object 80. Like the object map 50, the contact map 60 represents information referenced to determine at which position on the holding object 80 the worker should hold the holding object 80, assuming that the worker is holding the holding object 80. The contact map 60 is defined based on the shape of the contact portion of the end effector 2B with the holding object 80 and the shape of the holding object 80. The contact map 60 represents the appropriateness of the position at which the end effector 2B contacts the holding object 80. For example, the contact map 60 may define which parts should be holding positions and which parts should not be holding positions in relation to the shape or material of the contact portion of the end effector 2B with the holding object 80 and the shape or material of the holding position of the holding object 80. More specifically, a portion where the contact area between the end effector 2B and the object to be held 80 is small, a portion where the coefficient of friction between the end effector 2B and the object to be held 80 is smaller than a predetermined value, or other portion that is deemed difficult to hold for the end effector 2B to be used based on empirical rules, can be defined in the contact map 60 as a rule representing a portion that should not be a holding position. Similarly, a portion where the contact area between the end effector 2B and the object to be held 80 is large, a portion where the coefficient of friction between the end effector 2B and the object to be held 80 is larger than a predetermined value, or other portion that should be easy to hold for the end effector 2B to be used based on empirical rules, can be defined in the contact map 60 as a rule representing a portion that should be a holding position.

[0076] The contact map 60 represents, for example, the contact area between the surface of the holding object 80 and the fingers of the end effector 2B when the holding object 80 is held by the end effector 2B, or the frictional force acting between the surface of the holding object 80 and the fingers of the end effector 2B. If the surface of the holding object 80 has irregularities, the contact area may change significantly even if the position of the fingers of the end effector 2B is slightly shifted.

[0077] 20, the contact map 60 may be expressed in grayscale, with the outer periphery 62 of a cross section obtained by cutting the object to be held 80 along a plane at the height at which the object to be held is held, being expressed in a color closer to white, corresponding to a more appropriate holding position, and the farther from the center and closer to the corners, being expressed in a color closer to black, corresponding to an inappropriate holding position. The contact map 60 illustrated in FIG. 20 indicates that the contact area between the surface of the object to be held 80 and the fingers of the end effector 2B becomes larger when the object to be held is held in a position closer to the center of each side, and becomes smaller when the object to be held is held in a position closer to the corners.

[0078] 20, the solid black line surrounding the perimeter 62 of the contact map 60 is simply a line that represents the outline of the perimeter 62. It does not indicate that the coordinates where the solid black line is drawn are incorrect holding positions.

[0079] The control unit 12 may generate one contact map 60 that specifies multiple rules by combining contact maps 60 that specify one rule. When combining multiple contact maps 60, the control unit 12 may set a weighting coefficient for each contact map 60 before combining them. For example, if the contact area between the surface of the held object 80 and the fingers of the end effector 2B is important, the control unit 12 may set a large weighting coefficient for the contact map 60 that specifies the contact area.

[0080] Based on the generated rule map, the control unit 12 estimates the position at which the holding object 80 will be held. Specifically, the control unit 12 projects the opening width model 31 onto the rule map as a tentative holding position, and evaluates the appropriateness of actually holding the holding object 80 at the tentative holding position by calculating the degree of coincidence at the projected position.

[0081] The control unit 12 calculates the degree of match in the surrounding environment map 40 for each combination of the approach position 70 extracted along with the opening width of the end effector 2B and the rotation angle of the end effector 2B. Specifically, the control unit 12 projects the opening width model 31 onto the surrounding environment map 40 using the extracted approach position 70 and rotation angle. The control unit 12 calculates, as the degree of match, the average value of the numerical values ​​or color brightness set at each coordinate in the range where the approach region 41 and the opening width model 31 overlap in the surrounding environment map 40.

[0082] It can also be said that the position of the end effector 2B corresponding to the position and rotation angle of the opening width model 31 that can be projected onto the surrounding environment map 40 is a position that matches the surrounding environment map 40. It can also be said that the control unit 12 estimates the holding position from among the positions that match the surrounding environment map 40.

[0083] The control unit 12 calculates the degree of match in the object map 50 for each combination of the approach position 70 and the rotation angle of the end effector 2B extracted together with the opening width of the end effector 2B. Specifically, the control unit 12 projects the opening width model 31 onto the object map 50 using the extracted approach position 70 and rotation angle. The control unit 12 calculates, as the degree of match, the average value of the numerical values ​​or color brightness values ​​set at each coordinate in the range in the object map 50 where the cross section 52 of the held object 80 and the opening width model 31 overlap.

[0084] The object map 50 represents the suitability of a position for holding the holding target 80. The position of the end effector 2B corresponding to the position and rotation angle of the opening width model 31 projected onto the object map 50 can also be said to be a position that matches the object map 50. It can also be said that the control unit 12 estimates the holding position from among positions that match the object map 50.

[0085] The control unit 12 calculates the degree of match in the contact map 60 for each combination of the approach position 70 extracted along with the opening width of the end effector 2B and the rotation angle of the end effector 2B. Specifically, the control unit 12 projects the opening width model 31 onto the contact map 60 using the extracted approach position 70 and rotation angle. The control unit 12 calculates, as the degree of match, the average value of the numerical values ​​or color brightness values ​​set at each coordinate in the range where the periphery 62 in the contact map 60 overlaps with the opening width model 31.

[0086] The contact map 60 indicates the suitability of the contact portion of the end effector 2B with the holding object 80 as a position for contacting the holding object 80. It can also be said that the control unit 12 estimates the holding position from among multiple positions of the end effector 2B corresponding to the positions and rotation angles of the multiple opening width models 31 projected onto the contact map 60.

[0087] The control unit 12 may calculate the angle at which each of the holder positions 32 of the opening width model 31 is incident on the outer periphery 62 along the direction of the stroke range 34. In other words, the control unit 12 may calculate the angle of intersection at each of two points between the outer periphery 62 and a line along the direction of the stroke range 34 of the opening width model 31. In this embodiment, the control unit 12 calculates the angle of incidence as 0 degrees when the holder position 32 is perpendicular to the outer periphery 62. The control unit 12 may reflect the calculated angle in the value of the degree of match in the contact map 60. The control unit 12 may calculate a larger value for the degree of match as the angle approaches 0 degrees. The control unit 12 may calculate, for example, the degree of match by multiplying the average value of the brightness values ​​of the numbers or colors set at each coordinate in the range of the outer periphery 62 of the contact map 60 that overlaps with the projected opening width model 31 by the cosine value of the calculated angle.

[0088] If the outer periphery 62 has unevenness, the control unit 12 may generate a model in which the outer periphery 62 is flattened based on the thickness, width, or length of the holding portion of the end effector 2B, and calculate the angle at which the holding portion position 32 is incident on the flattened model.

[0089] The control unit 12 adds up the degrees of agreement calculated in each rule map for each combination of the approach position 70 and the rotation angle of the opening width model 31 to calculate an overall degree of agreement. The control unit 12 may weight the degrees of agreement calculated in each rule map and add them up. The control unit 12 assigns the same weight to the degrees of agreement calculated in each rule map for all combinations. The weighting coefficient applied to the degrees of agreement calculated in each rule map is also referred to as a map coefficient. A map coefficient may be determined for each map.

[0090] As described above, the control unit 12 calculates the overall degree of agreement for each holding position based on the opening width model 31 and the rule map. The overall degree of agreement corresponds to an appropriateness value that indicates the appropriateness of each holding position. Furthermore, the rule map indicates the appropriateness of a position for holding the object to be held 80 by using a numerical value or a color brightness value assigned to each position (each coordinate) of the rule map. The control unit 12 can calculate the appropriateness value by calculating the value assigned to each position when the end effector model 30 is overlaid on the rule map.

[0091] The control unit 12 compares the overall degree of agreement calculated for each combination of the approach position 70 and the rotation angle of the end effector model 30 and selects the combination with the highest overall degree of agreement. The control unit 12 estimates the position at which the fingers of the end effector 2B enter the holding object 80 when moving along the stroke direction, which is determined based on the approach position 70 and the rotation angle of the end effector model 30 of the selected combination, as the position at which the holding object 80 will be held. In other words, the control unit 12 estimates the holding position based on the end effector model 30 and the rule map. It can also be said that the control unit 12 estimates the holding position based on an appropriate value. The control unit 12 outputs the estimated holding position to the robot control device 110 via the interface 14.

[0092] (Example of procedure for estimating retention parameters) The control unit 12 of the retention parameter estimation device 10 may execute a retention parameter estimation method including the steps of the flowchart illustrated in Fig. 21. The retention parameter estimation method may be realized as a retention position estimation program executed by a processor constituting the control unit 12 of the retention parameter estimation device 10. The retention parameter estimation program may be stored in a non-transitory computer-readable medium.

[0093] The control unit 12 acquires data including information about the end effector 2B, a held object image obtained by capturing the held object 80, and depth data of the held object 80 (step S1). The control unit 12 generates an end effector model 30 (step S2). The control unit 12 estimates the height at which the held object 80 is held (step S3). The control unit 12 generates a mask image 20 (step S4).

[0094] The control unit 12 creates an approach map 90 and rule maps such as the surrounding environment map 40, the object map 50, and the contact map 60 (step S5). The control unit 12 projects the end effector model 30 onto the approach map 90 (step S6). The control unit 12 estimates the opening width of the end effector 2B by projecting the end effector model 30 onto the approach map 90 (step S7). The control unit 12 creates an opening width model 31 based on the estimated opening width (step S8). The control unit 12 projects the opening width model 31 onto each rule map (step S9). The control unit 12 calculates the degree of agreement between each rule map and the opening width model 31 projected onto each rule map (step S10). The control unit 12 weights the degree of agreement calculated for each rule map to calculate an overall degree of agreement (step S11). The control unit 12 selects the projection position of the opening width model 31 when the overall degree of coincidence is high, and estimates the position where the holding part of the end effector 2B enters the held object 80 at the selected position as the holding position (step S12). After executing the procedure of step S12, the control unit 12 ends the execution of the procedure of the flowchart in FIG.

[0095] The control unit 12 may perform the procedure of step S4 for generating the mask image 20 before step S2 or S3.

[0096] (Small summary) As described above, the holding parameter estimation device 10 according to this embodiment estimates the opening width when a portion of the end effector 2B is displaced to hold the holding object 80, based on the end effector model 30, the holding object image, and the depth data. If the opening width of the end effector 2B when displacing a portion of the end effector 2B toward the holding object 80 for holding is close to the width of the holding object 80, there is a risk of the end effector 2B colliding with the holding object during the displacement. On the other hand, if the opening width is too wide, it takes a long time to hold the holding object 80 after displacing a portion of the end effector 2B toward the holding object 80. Furthermore, if the opening width is too wide, there is a risk of the end effector 2B colliding with an obstacle around the holding object 80 when closing the end effector 2B for holding after the displacement. In response to such an event, the holding parameter estimation device 10 having the above-described configuration estimates the opening width during the displacement, and can therefore estimate an appropriate state when bringing the end effector 2B closer to the holding object 80. Therefore, the holding parameter estimation device 10 can provide parameters for operating the robot 2 so as to obtain an appropriate opening width.

[0097] Furthermore, in the holding parameter estimation device 10 of this embodiment, the approach region 91 in the approach map 90 is a region excluding a region obtained by expanding the presence region of the held object 80 and the first region 94 in the width direction of the holding portion of the end effector 2B by a size equal to or greater than half the width of the holding portion and in the thickness direction of the holding portion by a size equal to or greater than half the thickness of the holding portion. With this configuration, the holding parameter estimation device 10 can estimate the opening width within a region unlikely to interfere with the held object 80 and objects other than the held object 80.

[0098] Furthermore, the holding parameter estimation device 10 of this embodiment calculates a first center position CP1, which is the center position of the existence possibility region 99 for each holding part of the end effector 2B, and when the holding part is displaced from the first center position CP1 toward the holding target 80 along a predetermined direction for each holding part, an arbitrary position within the area through which the holding part passes before reaching the end of the existence possibility region 99, which does not interfere with the holding target 80 or objects other than the holding target 80, is estimated as the position of the holding part corresponding to the opening width in the existence possibility region 99. With this configuration, the holding parameter estimation device 10 can determine the presence or absence of interference at only one point at an arbitrary position, without determining the presence or absence of interference in the entire area of ​​the holding part. Therefore, the holding parameter estimation device 10 can quickly estimate the opening width.

[0099] Furthermore, in the holding parameter estimation device 10 of this embodiment, when the first center position CP1 deviates from the movable straight line AL, the control unit 12 estimates the position of the holding unit corresponding to the opening width by regarding the intersection of a perpendicular line drawn from the first center position CP1 to the movable straight line AL and the movable straight line AL as the first center position CP1. With this configuration, the holding parameter estimation device 10 can estimate the position of the holding unit corresponding to the opening width on the movable straight line AL even when a part of the held object 80 is located between the surface of the held object 80 at the innermost position within the movable range of the holding unit and the maximum opening width of the holding unit.

[0100] Furthermore, according to the holding parameter estimation device 10 of this embodiment, the holding position when the holding object 80 is held by the end effector 2B of the robot 2 is estimated based on a rule map. By estimating the holding position based on the rule map, the operator's experience, etc., can be reflected in the rule map. For example, when an operator holds various holding objects 80, the operator considers the position at which to hold each holding object 80. The operator determines the holding position by taking into account, for example, the center of gravity of the holding object 80, obstacles present around the holding object 80, or the wide range of positions for holding the holding object 80. In other words, according to the holding parameter estimation device 10 of this embodiment, the operator's intention is reflected in the rule map, the degree of agreement in each rule map is calculated, and the grip position is estimated based on the overall degree of agreement calculated by weighting the degree of agreement. As a result, the robot 2 can hold the holding object 80 at the holding position intended by the operator. In other words, the holding position of an object can be easily estimated to be a position that conforms to the operator's intention.

[0101] Furthermore, workers can use a rule map that defines rules for holding positions. As a result, learning is not required. Furthermore, by adding a rule map when a new rule emerges, it becomes easier to respond to changes in the environment. For example, a task that involves holding an object to be held 80 may include various components depending on the worker or the environment of the work site. Therefore, it is necessary to estimate and hold the holding position of the object to be held 80 based on each component. By generating a rule map that reflects various components and adding them to the objects to be calculated for the degree of coincidence, it becomes easier to respond to special rules, etc., for each worker or each site.

[0102] (Other embodiments) Other embodiments are described below.

[0103] <Use of estimation results based on other methods> The holding parameter estimation device 10 may acquire holding positions of the holding object 80 estimated based on another method. The holding parameter estimation device 10 may estimate the opening width for each of the acquired holding positions by executing the holding parameter estimation method according to the present embodiment on the acquired holding positions.

[0104] Specifically, the control unit 12 of the holding parameter estimation device 10 acquires, via the interface 14, a holding position of the holding object 80 estimated based on another technique. The control unit 12 calculates a combination of the approach position 70 and the rotation angle of the end effector model 30 corresponding to the acquired holding position. The control unit 12 projects the end effector model 30 onto the approach map 90 using the combination corresponding to the acquired holding position, and estimates the opening width. Furthermore, the control unit 12 may create an opening width model 31 based on the estimated opening width. The control unit 12 may project the opening width model 31 onto each rule map and calculate the degree of agreement for each rule map. The control unit 12 may weight and add up the degrees of agreement for each rule map to calculate an overall degree of agreement. The control unit 12 may select the opening width and holding position corresponding to the combination with the highest value of the overall degree of agreement and output the selected opening width and holding position to the robot control device 110.

[0105] The holding parameter estimation device 10 may calculate a comprehensive degree of agreement for the acquired holding position to evaluate the appropriateness of the acquired holding position. The control unit 12 evaluates whether the acquired holding position is appropriate based on the calculated comprehensive degree of agreement. The control unit 12 may determine that the acquired holding position is appropriate when, for example, the calculated comprehensive degree of agreement is equal to or greater than a predetermined value.

[0106] The holding parameter estimation device 10 may acquire the center position and rotation angle of the end effector 2B estimated based on another method. The holding parameter estimation device 10 can calculate a combination of the approach position 70 and the rotation angle by regarding the acquired center position of the end effector 2B as the approach position 70, and execute the holding parameter estimation method according to this embodiment.

[0107] The holding parameter estimation device 10 can reduce the number of combinations of the approach position 70 and the rotation angle to be used for calculating the degree of coincidence by acquiring the holding position of the holding object 80 estimated based on another method, thereby reducing the calculation load.

[0108] <Adjusting weights through learning> The retention parameter estimation device 10 calculates an overall degree of coincidence by weighting and summing the degrees of coincidence calculated for each rule map. The retention parameter estimation device 10 may update the weighting coefficients by learning based on annotation information for the estimated retention positions. The retention parameter estimation device 10 can improve the estimation accuracy of the retention positions by updating the weighting coefficients.

[0109] Specifically, the control unit 12 of the holding parameter estimation device 10 may notify the user of the estimated holding position via the interface 14. The control unit 12 may receive an input from the user to correct the holding position as an annotation to the estimated holding position via the interface 14. The control unit 12 may output the corrected holding position to the robot control device 110 based on the information of the correction by the user. The control unit 12 may update the weighting coefficient by learning based on the information of the correction by the user and re-estimate the holding position.

[0110] The control unit 12 may estimate a plurality of candidate holding positions and notify the user via the interface 14. The control unit 12 may estimate, as candidates, holding positions for which the overall degree of coincidence is equal to or greater than a predetermined value. The control unit 12 may receive, via the interface 14, an input from the user to select from the candidate holding positions as an annotation for the estimated holding position. The control unit 12 may output the holding position selected by the user to the robot control device 110. The control unit 12 may update the weighting coefficients by learning based on information selected by the user.

[0111] The control unit 12 may extract holding positions where the overall degree of agreement is equal to or greater than a predetermined value as candidate positions where the holding object 80 can be held, and notify the user of the extracted positions. The control unit 12 may receive, via the interface 14, an input from the user to modify the candidate positions or to select a candidate position as an annotation for the candidate positions. The control unit 12 may evaluate the suitability of each candidate position based on the input for modifying or selecting the candidate position. The control unit 12 may update the weighting coefficient for a candidate position that is evaluated as having a high suitability as a holding position based on the user's input, so that the overall degree of agreement is increased. The control unit 12 may output the selected candidate position to the robot control device 110 as the holding position. The control unit 12 may output the modified candidate position to the robot control device 110 as the holding position.

[0112] The control unit 12 of the retention parameter estimation device 10 may execute the procedure of the flowchart shown in Fig. 22. The control unit 12 selects candidates for retention positions by executing the procedure of the flowchart in Fig. 22, for example (step S21). The control unit 12 outputs the candidates for retention positions to the interface 14 (step S22). The control unit 12 corrects the retention positions based on a correction input by the user (step S23). The control unit 12 learns the correction content (step S24). The control unit 12 updates the weighting based on the learning result (step S25). After executing the procedure of step S25, the control unit 12 ends the execution of the procedure of the flowchart in Fig. 22.

[0113] As described above, the retention parameter estimation device 10 can update the weights based on the content of annotations made by the user, thereby improving robustness in adapting to various work environments.

[0114] The retention parameter estimation device 10 weights and blends multiple maps to generate a rule map. For example, the retention parameter estimation device 10 blends multiple object maps 50 to generate a single object map 50. When weighting and blending multiple maps, the retention parameter estimation device 10 may update the weighting coefficients for each map by learning based on annotations.

[0115] The holding parameter estimation device 10 may acquire annotation information for a plurality of holding positions and correct the appropriate value based on the annotation information.

[0116] <Image-based map generation> The holding parameter estimation device 10 may generate a rule map based on a holding target image captured by the camera 4. The control unit 12 of the holding parameter estimation device 10 may, for example, estimate the center of gravity of the holding target 80 based on the holding target image and generate an object map 50 that specifies the center of gravity position. The control unit 12 may, for example, estimate the material of the holding target 80 based on the holding target image and generate a contact map 60 that specifies the friction force acting between the surface of the holding target 80 and the holding portion of the end effector 2B. The control unit 12 may estimate the material of the holding target 80 based on the color or pattern of the holding target 80, or information regarding unevenness.

[0117] <Setting the height of the holding position> The holding parameter estimation device 10 first sets the height at which the holding object 80 is held, and estimates the opening width and holding position at the set height. The holding parameter estimation device 10 may change the height at which the holding object 80 is held, execute the holding parameter estimation method at each height, and estimate the combination of the holding height, approach position 70, and rotation angle at which the value of the overall degree of agreement is greatest, together with the opening width, as the holding position. This can improve the stability of holding.

[0118] The above has described an embodiment of the retention parameter estimation device 10. However, embodiments of the present disclosure can also be embodied as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card).

[0119] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0120] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0121] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0122] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0123] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0124] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, the configurations distinguished by descriptions such as "first" and "second" can have their numbers exchanged. For example, the first region can exchange the identifiers "first" and "second" with the second region. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0125] 2. Robot 2A Arm 2B End Effector 4. Camera 5 Robot movement range 6 Work starting platform 7 Work target table 10. Retention parameter estimator 12 Control Unit 14 Interface 20 Mask Images 22 Windows 24 Mask 30 End Effector Model 31 opening width model 32 Holding part position 34 stroke range 36 Operating range of the holding part 38 Out of Operating Range 40 Surrounding Area Map 41 Approach Area 42 Object Area 43 Non-approach area 44 First Area 50 Object Map 52 cross section 60 Contact Map 62 Outer circumference 70 Approach position 72a, 72b, 72c Projection Models 74a, 74c Projection position of holding part 80 Retention Objects 81 Main Unit 82 holding points 83 Bracket-shaped member 90 Approach Map 91 Approach Area 92 Object Area 93 Non-approach area 94 First Area 94ex Expanded area of ​​the first area 95 Boundary 96 Second Area 96ex Expanded area of ​​the object 97 Obstacle Area 98 External Area 99 Possible Existence Area 100 Robot Control System 110 Robot control device AL movable straight line CP1 1st center position C92 Center of object area SL: A straight line that passes through the center of the object area and intersects the outer edge of the obstacle area at one point. WP Position of the holding part corresponding to the opening width

Claims

1. an acquisition unit that acquires information about an end effector having a holding unit that holds a holding object with an arbitrary opening width, holding object information that indicates the holding object, and depth data about the holding object; a control unit that acquires an end effector model that indicates an area where the end effector may be present based on the information, and estimates an opening width of the holding unit to hold the object to be held based on the end effector model, the holding object information, and the depth data; The retention target information is information indicating the area of ​​the retention target. Retention parameter estimator.

2. 2. The retention parameter estimation device according to claim 1, The control unit creating an approach map indicating an approach area in which the end effector can be opened without interfering with an object other than the held object at a height at which the held object is held, based on the held object information and the depth data; The opening width is estimated based on the approach area and an area where the end effector may be present. Retention parameter estimator.

3. 3. The retention parameter estimation device according to claim 2, The approach area is an area around the object to be held where the end effector can be opened without interfering with the object to be held, excluding an area where the object to be held is present and a first area facing the opposite side to the object to be held from an outer edge facing the object to be held in an area where an object other than the object to be held is present. Retention parameter estimator.

4. 4. The retention parameter estimation device according to claim 3, the end effector has a holding portion that is brought into contact with the object to be held when holding the object to be held, The approach area is an area obtained by expanding the area where the object to be held is present and the first area in the width direction of the holding portion of the end effector by a size equal to or greater than half the width of the holding portion and in the thickness direction of the holding portion by a size equal to or greater than half the thickness of the holding portion. Retention parameter estimator.

5. 5. The retention parameter estimation device according to claim 2, The control unit In superimposing the end effector model on the approach map, a possible area where the approach area and the end effector model overlap is estimated for each holding part of the end effector; An arbitrary point in the possible presence area for each of the holding parts is estimated as the position of the holding part corresponding to the opening width in the possible presence area. Retention parameter estimator.

6. 6. The retention parameter estimation device according to claim 5, The holding portion is configured to be movable along a predetermined direction, The control unit calculating a first center position that is a center position of the possible presence area for each holding portion of the end effector; When the holding portions are displaced from the first center position toward the holding object along the predetermined direction for each of the holding portions, any position within the area through which the holding portions pass before reaching the edge of the possible existence area that does not interfere with the holding object and objects other than the holding object is estimated as the position of the holding portion corresponding to the opening width in the possible existence area. Retention parameter estimator.

7. 7. The retention parameter estimation device according to claim 6, When the first center position deviates from a movable straight line that passes through a center position of an area in the end effector model where the end effector can be present and is parallel to the opening and closing direction of the holding unit, the control unit regards an intersection of a perpendicular line drawn from the first center position to the movable straight line and the movable straight line as the first center position, and estimates the position of the holding unit corresponding to the opening width. Retention parameter estimator.

8. 7. The retention parameter estimation device according to claim 6, The control unit displaces the end effector from the first center position toward a center position of a region in the end effector model where the end effector can exist, and if the approach region does not exist, searches for the approach region in the region where the end effector can exist outside the first center position. Retention parameter estimator.

9. 5. The retention parameter estimation device according to claim 2, The holding portion of the end effector is formed by at least two members that are movable opposite to each other. Retention parameter estimator.

10. 5. The retention parameter estimation device according to claim 1, The control unit creating an opening width model that identifies an area within the opening width where the end effector may be present; generating a rule map including a map that defines an area of ​​the object to be held by the end effector based on the object to be held; A position where the end effector comes into contact with the object to be held is estimated as a holding position based on the opening width model and the rule map. Retention parameter estimator.

11. 11. The retention parameter estimation device according to claim 10, The control unit calculating appropriateness values ​​representing appropriateness of the holding positions for the plurality of holding positions based on the opening width model and the rule map; The holding position is estimated based on the appropriate value. Retention parameter estimator.

12. 12. The retention parameter estimation device according to claim 11, the opening width model and the rule map are expressed by values ​​indicating suitability of each assigned position as a position for holding the object to be held, The control unit calculates the appropriate value by calculating a value assigned to each position when the opening width model is superimposed on the rule map. Retention parameter estimator.

13. 12. The retention parameter estimation device according to claim 11, The control unit calculates the appropriate value based on the rule map based on the appropriate values ​​based on a plurality of the maps included in the rule map and map coefficients determined for each of the maps. Retention parameter estimator.

14. Acquire information about an end effector having a holding portion that holds a holding object, holding object information indicating the holding object, and depth data about the holding object; obtaining an end effector model that indicates a region where the end effector may be present based on the information; estimating an opening width of the holding portion for holding the object to be held based on the end effector model, the holding object information, and the depth data; The retention target information is information indicating the area of ​​the retention target. Retention parameter estimation methods.

Images