Holding position determining device and holding position determining method

Abstract

This hold position determination device comprises a control unit that determines, as a hold position, a position where an end effector holding a held object is in contact with the held object. The control unit acquires an end effector model that identifies a region where a holding unit of the end effector may be present. On the basis of a held object image in which the held object is captured, the control unit acquires a rule map including a map that defines the position of the held object to be held by the end effector. The control unit determines the holding position on the basis of the end effector model and the rule map.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2021-90521 (filed May 28, 2021), the entire disclosure of which is incorporated herein by reference. [Technical field]

[0002] The present disclosure relates to a holding position determining device and a holding position determining method. [Background technology]

[0003] 2. Description of the Related Art There is known a learning device that performs learning to appropriately determine a position and a posture for grasping an object from an image on a computer (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-205929 A Summary of the Invention

[0005] A holding position determination device according to an embodiment of the present disclosure includes a control unit that determines a position where an end effector holding a holding object comes into contact with the holding object as a holding position. The control unit acquires an end effector model that specifies an area where a holding part of the end effector may be present. The control unit acquires a rule map including a map that specifies a position of the holding object to be held by the end effector based on a holding object image obtained by photographing the holding object. The control unit determines the holding position based on the end effector model and the rule map.

[0006] A holding position determination method according to an embodiment of the present disclosure is executed by a holding position determination device that determines a position where an end effector holding a holding object comes into contact with the holding object as a holding position. The holding position determination method includes the holding position determination device acquiring an end effector model that specifies an area where a holding portion of the end effector may be present. The holding position determination method includes the holding position determination device acquiring a rule map including a map that specifies a position of the holding object to be held by the end effector based on a holding object image obtained by photographing the holding object. The holding position determination method includes the holding position determination device determining the holding position based on the end effector model and the rule map. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Diagram 2] 11 is a side view showing an example in which an object to be held is held by an end effector. FIG. [Diagram 3] 1 is a block diagram showing an example of the configuration of a robot control system according to an embodiment. [Figure 4] 1 is a diagram showing an example of a mask image representing the outline of an object to be held on a plane on which the object to be held is held; FIG. [Figure 5A] FIG. 13 is a diagram showing an example of a model representing the opening width of a finger in the end effector model. [Figure 5B] FIG. 13 is a diagram showing an example of a model representing a finger stroke in the end effector model. [Figure 5C] FIG. 13 is a diagram showing an example of an entire model of the end effector model, which combines the finger opening width and stroke. [Figure 6] FIG. 2 is a diagram illustrating an example of a surrounding environment map. [Figure 7] 11A and 11B are diagrams illustrating a method for generating an approach range of a surrounding environment map. [Figure 8A] FIG. 13 is a diagram showing an example of an object map showing the center of a holding object. [Figure 8B] FIG. 13 is a diagram showing an example of an object map indicating the priority of holding positions designated by a user. [Figure 9] FIG. 13 is a diagram illustrating an example of a contact map. [Figure 10] 11A and 11B are diagrams illustrating examples of positions at which an end effector model is projected onto a surrounding environment map. [Figure 11] 11 is a diagram showing positions where an end effector model can be projected onto a surrounding environment map and positions where it cannot be projected onto the surrounding environment map. FIG. [Figure 12A] 13A and 13B are diagrams illustrating examples of positions at which an end effector model is projected onto an object map. [Figure 12B] 13A and 13B are diagrams illustrating examples of positions at which an end effector model is projected onto a contact map. [Figure 13] 13 is a flowchart showing an example of a procedure of a holding position determining method. [Figure 14] 14 is a flowchart showing an example of a procedure for correcting and learning a holding position determined by the holding position determination method of FIG. 13; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] When learning the holding position of an object, a large amount of training data is required so that the holding position of the object is a position that conforms to human intention. The workload or cost of preparing a large amount of training data increases. According to the holding position determination device and holding position determination method according to an embodiment of the present disclosure, the holding position of the object can easily conform to human intention.

[0009] (Configuration example of robot control system 100) 1, 2, and 3, a robot control system 100 according to an embodiment of the present disclosure includes a robot 2, a camera 4, a robot control device 110, and a holding position determination device 10. The robot 2 holds a holding target 80 with an end effector 2B to perform a task. The robot control device 110 controls the robot 2. The holding position determination device 10 determines a position where the robot 2 comes into contact when holding the holding target 80 as a holding position, and outputs the position to the robot control device 110.

[0010] In this embodiment, the robot 2 holds the holding object 80 on the work start table 6. That is, the robot control device 110 controls the robot 2 to hold the holding object 80 on the work start table 6. The robot 2 may move the holding object 80 from the work start table 6 to the work target table 7. The holding object 80 is also referred to as a work target. The robot 2 operates inside the operating range 5.

[0011] <Robot 2> The robot 2 includes an arm 2A and an end effector 2B. The arm 2A may be configured, for example, as a 6-axis or 7-axis vertical multi-joint robot. The arm 2A may be configured as a 3-axis or 4-axis horizontal multi-joint robot or a SCARA robot. The arm 2A may be configured as a 2-axis or 3-axis Cartesian robot. The arm 2A may be configured as a parallel link robot or the like. The number of axes constituting the arm 2A is not limited to those illustrated. In other words, the robot 2 has an arm 2A connected by a plurality of joints, and operates by driving the joints.

[0012] The end effector 2B may include, for example, a gripper configured to hold the holding object 80. The gripper may have at least one finger. The gripper finger may have one or more joints. The gripper finger may have an adsorption portion that holds the holding object 80 by adsorption. The end effector 2B may be configured as one finger having an adsorption portion. The end effector 2B may be configured as two or more fingers that pinch and hold the holding object 80. The end effector 2B may include a scooping hand configured to scoop the holding object 80. The end effector 2B is not limited to these examples and may be configured to perform various other operations. In the configuration illustrated in FIG. 1, the end effector 2B includes a gripper. A portion where the end effector 2B holds the holding object 80 is also referred to as a holding portion. The holding portion of the end effector 2B may include a portion that contacts the holding object 80.

[0013] The robot 2 can control the position of the end effector 2B by operating the arm 2A. The end effector 2B may have an axis that is a reference for the direction of the action on the held object 80. If the end effector 2B has an axis, the robot 2 can control the direction of the axis of the end effector 2B by operating the arm 2A. The robot 2 controls the start and end of the action of the end effector 2B acting on the held object 80. The robot 2 can move or process the held object 80 by controlling the action of the end effector 2B while controlling the position of the end effector 2B or the direction of the axis of the end effector 2B. In the configuration exemplified in FIG. 1, the robot 2 makes the end effector 2B hold the held object 80 at the work start table 6 and moves the end effector 2B to the work target table 7. The robot 2 makes the end effector 2B release the held object 80 at the work target table 7. In this manner, the robot 2 can move the object to be held 80 from the work start table 6 to the work target table 7.

[0014] <Camera 4> In the configuration example shown in FIG. 1, the robot control system 100 includes a camera 4 attached to the end effector 2B of the robot 2. The camera 4 photographs the holding object 80 from the end effector 2B toward the holding object 80. That is, the camera 4 photographs the holding object 80 from the direction in which the end effector 2B holds the holding object 80. The camera 4 may be configured to photograph the holding object 80 from another direction. An image of the holding object 80 photographed from another direction may be converted into an image photographed from the direction in which the end effector 2B holds the holding object 80. The image of the holding object 80 is also referred to as a holding object image. The camera 4 includes a depth sensor and is configured to be able to acquire depth data of the holding object 80. The image photographed by the camera 4 may include monochrome luminance information, or may include luminance information of each color expressed by RGB (Red, Green and Blue), etc. The number of cameras 4 is not limited to one, and may be two or more. The camera 4 may also capture images of other objects located within a predetermined range from the held object 80 as obstacles, and obtain depth data of the obstacles.

[0015] <Holding position determining device 10> As shown in FIG. 3, the holding position determination device 10 includes a control unit 12 and an interface 14. The interface 14 obtains information or data related to the holding object 80 from an external device and outputs the information or data to the external device. The interface 14 also obtains an image of the holding object 80 from the camera 4. The interface 14 may receive input from a user. The interface 14 may output information or data so that the user can recognize it. The control unit 12 determines a position where the robot 2 holds the holding object 80 based on the information or data obtained by the interface 14, and outputs the information or data to the interface 14. The position where the robot 2 holds the holding object 80 is also simply referred to as a holding position.

[0016] The control unit 12 may include at least one processor to provide control and processing power for executing various functions. The processor may execute programs that realize the various functions of the control unit 12. The processor may be realized as a single integrated circuit. The integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as multiple communicatively connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.

[0017] The control unit 12 may include a memory unit. The memory unit may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The memory unit stores various information. The memory unit stores programs executed by the control unit 12, etc. The memory unit may be configured as a non-transitory readable medium. The memory unit may function as a work memory for the control unit 12. At least a part of the memory unit may be configured as a separate entity from the control unit 12.

[0018] The interface 14 may include a communication device configured to be capable of wired or wireless communication. The communication device may be configured to be capable of communication using a communication method based on various communication standards. The communication device may be configured using known communication technology.

[0019] The interface 14 may be configured to include an input device that accepts input of information or data from a user. The input device may be configured to include, for example, a touch panel or a touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include a voice input device such as a microphone.

[0020] The interface 14 includes an output device that outputs information or data to a user. The output device may include, for example, a display device that outputs visual information such as an image, text, or graphics. The display device may include, for example, a Liquid Crystal Display (LCD), an organic EL (Electro-Luminescence) display, an inorganic EL display, or a Plasma Display Panel (PDP). The display device is not limited to these displays and may include displays of various other types. The display device may include a light-emitting device such as an LED (Light Emission Diode) or an LD (Laser Diode). The display device may include various other devices. The output device may include, for example, an audio output device such as a speaker that outputs audio information such as voice. The output device is not limited to these examples and may include various other devices.

[0021] <Robot control device 110> The robot control device 110 acquires information for identifying the holding position from the holding position determination device 10 and controls the robot 2 so that the robot 2 holds the object to be held 80 at the holding position determined by the holding position determination device 10 .

[0022] The robot controller 110 may be configured to include at least one processor to provide control and processing power for executing various functions. Each component of the robot controller 110 may be configured to include at least one processor. A plurality of components among the components of the robot controller 110 may be realized by one processor. The entire robot controller 110 may be realized by one processor. The processor may execute programs that realize various functions of the robot controller 110. The processor may be configured the same as or similar to the processor used in the holding position determination device 10.

[0023] The robot control device 110 may include a memory unit. The memory unit may be configured in the same manner as or similar to the memory unit used in the holding position determination device 10.

[0024] The robot control device 110 may include the holding position determination device 10. The robot control device 110 and the holding position determination device 10 may be configured as separate entities.

[0025] (Operation example of robot control system 100) The robot control system 100 controls the robot 2 by the robot control device 110 to cause the robot 2 to perform a task. In this embodiment, the task to be performed by the robot 2 includes an operation of holding an object to be held 80. In the robot control system 100, a holding position determination device 10 determines a holding position for the object to be held 80 by the robot 2. The robot control device 110 controls the robot 2 so that the robot 2 holds the object to be held 80 at the determined holding position.

[0026] When the end effector 2B is a gripper, the control unit 12 determines, as the holding position, a combination of positions where the fingers come into contact with the holding object 80 when the gripper grasps the holding object 80. When the end effector 2B has an adsorption unit, the control unit 12 determines, as the holding position, a position where the adsorption unit comes into contact with the holding object 80 when the end effector 2B adsorbs the holding object 80.

[0027] The control unit 12 of the holding position determination device 10 acquires an image of the holding object 80 and depth data of the holding object 80 from the camera 4. The control unit 12 recognizes the outer shape and position of the holding object 80 based on the image of the holding object 80 and the depth data. As illustrated in FIG. 4, the control unit 12 generates a mask image 20 representing the recognition result of the holding object 80 when the holding object 80 is viewed from the camera 4 attached to the end effector 2B. The mask image 20 includes a window 22 representing the area where the holding object 80 exists as viewed from the camera 4, and a mask 24 representing the other area. The window 22 is represented as a white area. The mask 24 is represented as an area hatched with diagonal lines slanting upward to the right. Note that the mask 24 in the mask image 20 is represented as a hatched area for convenience of illustration, but may be represented as a black area in an actual embodiment.

[0028] The control unit 12 acquires information about the end effector 2B. For example, when the end effector 2B is a gripper, the information about the end effector 2B includes information specifying the maximum interval between gripping fingers, the thickness of the fingers, the width of the fingers, and the like. The control unit 12 may generate an end effector model 30 based on the information about the end effector 2B. The control unit 12 may acquire the end effector model 30 that specifies an area where the holding part of the end effector 2B may be present.

[0029] As shown in FIG. 5A, the end effector model 30 includes a finger model that specifies a finger position 32 representing a range in which the fingers of the gripper are positioned with a predetermined distance therebetween and a non-operating range 38 representing a range other than the finger position 32. That is, the finger model represents the opening width of the fingers of the gripper. The finger position 32 may represent a range in which the fingers of the gripper are positioned with a maximum distance therebetween. In this case, the finger model represents the maximum opening width of the fingers of the gripper. It is assumed that the finger model shown in FIG. 5A represents the maximum opening width of the fingers of the gripper. The finger position 32 is represented as a white area. The non-operating range 38 is represented as a hatched area with diagonal lines slanting upwards to the right. Note that the non-operating range 38 is represented as a hatched area for convenience of illustration in the drawings, but may be represented as a black area in an actual embodiment.

[0030] The end effector model 30 includes a stroke model that specifies a stroke range 34 that represents the range over which the gripper fingers operate, as shown in Figure 5B. The stroke range 34 is represented as a white filled area.

[0031] As shown in FIG. 5C, the end effector model 30 includes an entire model that combines the finger model of FIG. 5A and the stroke model of FIG. 5B. The entire model specifies a finger motion range 36. The finger motion range 36 includes a finger position 32. In FIG. 5C, the finger portion of the gripper's finger motion range 36 is shown by a dashed line, but in an actual embodiment, it may not be distinguished. In the present embodiment, the end effector model 30 represents a range in which the gripper's fingers are positioned with a maximum gap (maximum opening width), but is not limited to this. The end effector model 30 may represent a range in which the gripper's fingers are positioned with an arbitrary gap (predetermined gap). For example, the end effector model 30 may represent a gap between the gripper's fingers according to the size of the object that the end effector 2B is to grip.

[0032] The control unit 12 may generate only the whole model as the end effector model 30. In this case, the finger position 32 may be identified by associating information identifying the finger position 32 with the whole model. The information identifying the finger position 32 may include numerical values ​​representing the feature points of the finger.

[0033] When the end effector 2B has an adsorption portion, the end effector model 30 is configured as a model that defines the range in which the adsorption portion interferes with other objects when adsorbing the object 80 to be held.

[0034] The control unit 12 sets the height of the position where the holding object 80 is held based on the depth data of the holding object 80. Specifically, as illustrated in FIG. 2, the control unit 12 sets the height from the work start table 6 as the height of the position where the holding object 80 placed on the work start table 6 is held. In the case of FIG. 2, the position where the holding object 80 is pinched and held by the fingers of the end effector 2B is represented as a holding point 82. The height of the position of the holding point 82 from the work start table 6 is represented as H. In this case, the control unit 12 sets H as the height of the position where the holding object 80 is held. Based on the depth data of the holding object 80, the control unit 12 sets the height of the position where the holding object 80 is held to a value smaller than the distance from the work start table 6 to the highest point of the holding object 80. The control unit 12 may set the height of the position where the holding object 80 is held to a value approximately half the height of the holding object 80.

[0035] The control unit 12 may generate the mask image 20 based on the height of the position where the held object 80 is held and depth data of the held object 80. Specifically, the control unit 12 may generate the mask image 20 in which the cross-sectional shape of the held object 80 in a plane at the height of the position where the held object 80 is held is defined as a window 22.

[0036] The control unit 12 acquires a rule map that specifies a rule for determining a holding position of the holding object 80. The rule map includes a map that specifies a position of the holding object 80 that the end effector 2B should use for holding. The rule map may include a map that is generated based on the height of a position at which the holding object 80 is held. The rule map may be generated based on a holding object image. The rule map may be generated based on an image that is obtained by converting a holding object image that is taken of the holding object 80 from a direction other than the direction in which the end effector 2B holds the holding object 80 into an image taken from the direction in which the end effector 2B holds the holding object 80. The rule map may be classified into, for example, a surrounding environment map 40 (see FIG. 6), an object map 50 (see FIG. 8A or FIG. 8B), or a contact map 60 (see FIG. 9). The control unit 12 may generate the rule map. The control unit 12 may acquire a rule map that is based on at least one of shape data of the holding object 80 and depth data associated with the holding object image. The rule map may be acquired for each height of the position at which the holding object 80 is held. The rule map may specify rules according to the display form of an image expressed two-dimensionally.

[0037] As shown in FIG. 6, the surrounding environment map 40 specifies an object region 42, an obstacle region 44, an approach region 46, and a non-approach region 48. The object region 42 represents a range in which the holding object 80 exists at the height of the position where the holding object 80 is held. The obstacle region 44 represents a range in which an obstacle that is an object other than the holding object 80 and limits the movement of the end effector 2B exists at the height of the position where the holding object 80 is held. The approach region 46 represents a movement range that combines the movement of the gripper itself as the end effector 2B and the movement of the fingers of the gripper. The object region 42, the obstacle region 44, and the non-approach region 48 represent ranges other than the approach region 46. The approach region 46 is represented as a white region. The object region 42, the obstacle region 44, and the non-approach region 48 are represented as regions with diagonal hatching that slopes upward to the right. The object region 42, the obstacle region 44, and the non-approach region 48 are shown as hatched regions for convenience of illustration in the drawings, but may be shown as black regions in an actual embodiment. The surrounding environment map 40 is configured as a map that defines a range in which the end effector 2B can move at the height of the position where the held object 80 is held.

[0038] The control unit 12 can generate the surrounding environment map 40 by generating an approach region 46 and a non-approach region 48, and arranging the object region 42 and the obstacle region 44 in the approach region 46. The approach region 46 is also referred to as a first region in which the end effector 2B can be present. The non-approach region 48 is also referred to as a second region in which the end effector 2B cannot be present. The second region is defined as at least the region in which the held object 80 is present and the region around it. In the region around the held object 80, the second region is allocated such that the proportion of the second region in the first region decreases with increasing distance from the held object 80.

[0039] The control unit 12 can generate the approach region 46 based on the mask image 20. For example, a method of the control unit 12 generating the approach region 46 will be described with reference to FIG. 7. The control unit 12 may execute convolution of the end effector model 30 with respect to the mask image 20. The control unit 12 may move the finger movement range 36 so that at least a part of the finger movement range 36 specified by the end effector model 30 overlaps with the window 22 included in the mask image 20. In this case, the rectangle representing the finger movement range 36 may be rotated and positioned at various angles so that at least a part of the rectangle overlaps with the point at the upper left corner of the window 22, for example. The control unit 12 generates the region through which the finger movement range 36 passes when the finger movement range 36 is moved as the approach region 46. In FIG. 7, the trajectory drawn by the point farthest from the window 22 when the finger movement range 36 moves is represented as the boundary 46a of the approach region 46. The boundary 46a is represented by a dashed line.

[0040] The control unit 12 generates an object region 42 in the generated approach region 46 based on depth data of the held object 80. The control unit 12 generates a range in which the held object 80 exists at a position higher than the height of the position where the held object 80 is held as the object region 42. The control unit 12 also places an obstacle region 44 in the generated approach region 46 based on depth data of the obstacle. The control unit 12 generates a range in which an obstacle exists at a position higher than the height of the position where the held object 80 is held as the obstacle region 44.

[0041] The control unit 12 may execute a blurring process for the generated surrounding environment map 40 to blur the boundaries. Each area included in the surrounding environment map 40 generated by the above-described procedure can be distinguished as a numerical value at each coordinate in the map. For example, the control unit 12 may set a numerical value at a coordinate included in the approach area 46 to 1, which indicates that the point specified by the coordinate is included in the operating range of the end effector 2B. On the other hand, the control unit 12 may set a numerical value at a coordinate included in the object area 42, the obstacle area 44, and the non-approach area 48 to 0, which indicates that the point specified by the coordinate is not included in the operating range of the end effector 2B. The control unit 12 sets a value, for example, 0.5, that is greater than 0 and smaller than 1, for the numerical value at the coordinate that specifies a point within a predetermined range from the boundary between the area where 1 is set and the area where 0 is set. The control unit 12 executes the process of blurring the boundaries of the areas as a blurring process.

[0042] Moreover, the surrounding environment map 40 generated by the above-mentioned procedure can be distinguished by the color at each coordinate in the map. For example, the control unit 12 may represent points included in the approach area 46 in white, and represent points included in other areas in black. The control unit 12 may represent the color of points within a predetermined range from the boundary between the area represented by white and the area represented by black in grayscale. The control unit 12 may execute a process of blurring the boundaries of the areas in this way as a blurring process. Representing the color of each area in black, white, and gray is equivalent to representing the numerical value set for each area as a brightness value.

[0043] The control unit 12 performs the blurring process, thereby reducing the possibility that the end effector 2B will unexpectedly collide with an object due to an error in the operating range of the end effector 2B or an error in the contour of the held object 80 or an obstacle. In other words, the control unit 12 can determine the position of the end effector 2B relative to the held object 80 by taking various margins into consideration through the blurring process. The blurring process may be performed on the periphery of each area included in the generated surrounding environment map 40. Each area is enlarged by the blurring process.

[0044] The object map 50 represents information to be referred to in determining at what position of the holding object 80 the worker should hold the holding object 80 when the worker holds the holding object 80. The object map 50 represents information such as the shape, material, or density distribution of the holding object 80, for example.

[0045] For example, it is assumed that there is a holding object 80 whose holding position is more appropriate in accordance with the rules, the closer to the center of the holding object 80. As shown in FIG. 8A, the object map 50 for the holding object 80 assumed in this way may be expressed in a grayscale in which the closer to the center of the holding object 80 is, the closer to the rule the holding position is, the more appropriate it is to the rules, and the farther away from the center is, the closer to the black the holding position is, the more inappropriate it is to the rules. Also, as shown in FIG. 8B, the object map 50 may be expressed in a grayscale in which the user arbitrarily sets a rule for determining the position to hold the holding object 80. The object map 50 may be represented by a gray scale that specifies the rule that is applied to the object to be held 80. When creating an object map 50 in which the proper holding positions are determined as the object approaches the center of the object to be held 80 as shown in Fig. 8B, the object to be held 80 is represented in a color closer to white, which corresponds to a proper holding position, and the object to be held 80 is represented in a color closer to black, which corresponds to an inappropriate holding position, as the object approaches the center along the horizontal direction of the cross section 52 of the object to be held 80, and the object to be held 80 is represented in a color closer to black, which corresponds to an inappropriate holding position. Also, the object map 50 in Fig. 8B is represented in the same color along the height direction of the cross section 52.

[0046] 8A and 8B, the solid black lines surrounding the cross sections 52 of the object map 50 are lines that simply represent the outline of the cross sections 52 and do not indicate rules. They do not indicate that the coordinates where the solid black lines are drawn are incorrect holding positions.

[0047] The object map 50 may be configured such that, when assuming a position for holding the object 80 and evaluating the suitability of holding the object at the assumed position, an aptitude value representing the evaluation of suitability is increased by holding the object near an area represented by a color close to white. The object map 50 may also be represented by associating a numerical value with the color of each coordinate, as in the surrounding environment map 40. The control unit 12 may, for example, set 1 to the coordinate represented by white and set 0 to the coordinate represented by black.

[0048] The control unit 12 may generate the object map 50 in which a color or a numerical value is set for each coordinate so as to specify various rules, without being limited to the above example. For example, the control unit 12 may generate the object map 50 in which a color or a numerical value is set for each coordinate according to the distance from the center of gravity of the holding target 80. For example, the control unit 12 may generate the object map 50 in which a color or a numerical value is set for each coordinate so as to specify a position to be avoided as a holding position or a position to be prohibited as a holding position.

[0049] The control unit 12 may generate one object map 50 that specifies a plurality of rules by mixing object maps 50 that specify one rule. When mixing a plurality of object maps 50, the control unit 12 may mix the object maps 50 by setting a weighting coefficient for each object map 50. For example, when the position of the center of gravity of the held object 80 is important, the control unit 12 may set a large weighting coefficient for the object map 50 that specifies the position of the center of gravity.

[0050] The object map 50 may be defined based on the characteristics of the holding object 80 itself. The object map 50 may be defined based on any of the shape, material, texture, weight, or friction coefficient of the holding object 80. The object map 50 may be based on a rule arbitrarily defined by a user for the holding position of the holding object 80. For example, parts of the holding object 80 that should not be used as a holding position for various reasons, such as parts that are easily damaged or deformed by contact, parts with grease or the like attached, and parts that are slippery and not suitable for holding, may be defined as rules in the object map 50. Similarly, parts that are unlikely to be damaged or deformed, parts that are not grease or the like attached, parts that are not slippery, and other parts that should be used as a holding position based on empirical rules (parts that are considered to be easy to hold) may be defined as rules in the object map 50. By expressing the object map 50 as a two-dimensional image, it becomes easy to understand what rules are defined for the holding position of the holding object 80. The object map 50 may be generated for each type of holding object 80.

[0051] The contact map 60 represents a rule determined based on the relationship between the finger of the end effector 2B and the surface state of the holding object 80. Like the object map 50, the contact map 60 represents information to be referenced in order to determine at which position of the holding object 80 the worker holds the holding object 80, assuming that the worker holds the holding object 80. The contact map 60 is defined based on the shape of the contact portion of the end effector 2B with the holding object 80 and the shape of the holding object 80. The contact map 60 represents the suitability of the position at which the end effector 2B contacts the holding object 80. For example, the contact map 60 may define a portion that should be the holding position or a portion that should not be the holding position in relation to the shape or material of the contact portion of the end effector 2B with the holding object 80 and the shape or material of the holding position of the holding object 80. More specifically, a portion where the contact area between the end effector 2B and the held object 80 is small, a portion where the friction coefficient between the end effector 2B and the held object 80 is smaller than a predetermined value, or other portions that are considered to be difficult to hold for the end effector 2B used based on empirical rules can be defined in the contact map 60 as rules representing portions that should not be held positions. Similarly, a portion where the contact area between the end effector 2B and the held object 80 is large, a portion where the friction coefficient between the end effector 2B and the held object 80 is larger than a predetermined value, or other portions that should be easy to hold for the end effector 2B used based on empirical rules can be defined in the contact map 60 as rules representing portions that should be held positions.

[0052] The contact map 60 represents, for example, the contact area between the surface of the object to be held 80 and the fingers of the end effector 2B when the object to be held 80 is held by the end effector 2B, or the frictional force acting between the surface of the object to be held 80 and the fingers of the end effector 2B. If the surface of the object to be held 80 has irregularities, the contact area may vary significantly even if the position of the fingers of the end effector 2B is shifted slightly.

[0053] The contact map 60 may be expressed in a gray scale, as shown in Fig. 9, in which the outer periphery 62 of a cross section obtained by cutting the object to be held 80 along a plane at the height of the position at which the object to be held 80 is held is expressed in a color closer to white, which corresponds to a more appropriate holding position, and the farther away from the center and closer to the corners is, the more inappropriate the holding position is expressed in a color closer to black, which corresponds to a less appropriate holding position. The contact map 60 illustrated in Fig. 9 indicates that the contact area between the surface of the object to be held 80 and the fingers of the end effector 2B becomes large when the object to be held is held in a position close to the center of each side, and the contact area becomes small when the object to be held is held in a position close to the corners.

[0054] 9, the solid black line surrounding the perimeter 62 of the contact map 60 is simply a line that represents the outline of the perimeter 62. It does not represent that the coordinates where the solid black line is drawn are incorrect holding positions.

[0055] The control unit 12 may generate one contact map 60 that specifies a plurality of rules by mixing contact maps 60 that specify one rule. When mixing a plurality of contact maps 60, the control unit 12 may mix the contact maps 60 by setting a weighting coefficient for each contact map 60. For example, when the contact area between the surface of the held object 80 and the fingers of the end effector 2B is important, the control unit 12 may set a large weighting coefficient for the contact map 60 that specifies the contact area.

[0056] Based on the generated rule map, the control unit 12 determines a position for holding the holding object 80. Specifically, the control unit 12 projects the end effector model 30 onto the rule map as a tentative holding position, and evaluates the suitability of actually holding the holding object 80 at the tentative holding position by calculating the degree of coincidence at the projected position.

[0057] The control unit 12 selects an arbitrary point within the range of the window 22 of the mask image 20. The arbitrary point selected within the range of the window 22 corresponds to an approach position 70 included in the object region 42 in the surrounding environment map 40, as shown in FIG. 10. That is, the control unit 12 sets an arbitrary point within the range of the window 22 of the mask image 20 as the approach position 70. The control unit 12 projects the end effector model 30 with the center aligned with the approach position 70. The projected end effector model 30 is represented as projection models 72a and 72b in FIG. 10. The projection model 72a corresponds to a model obtained by rotating the end effector model 30 along the short side direction of the object region 42. The projection model 72b corresponds to a model obtained by rotating the end effector model 30 45 degrees clockwise from the short side direction of the object region 42.

[0058] The control unit 12 determines that the end effector model 30 cannot be projected when the finger position 32 included in the end effector model 30 overlaps the object region 42. As shown in FIG. 11, the projection position 74a of the finger included in the projection model 72a does not overlap the object region 42. Therefore, the control unit 12 can project the projection model 72a onto the surrounding environment map 40. On the other hand, the projection position 74c of the finger included in the projection model 72c overlaps the object region 42. Therefore, the control unit 12 cannot project the projection model 72c onto the surrounding environment map 40. By not projecting the end effector model 30 when the finger position 32 overlaps the object region 42, it is possible to prevent the end effector 2B from colliding with the held object 80 while the end effector 2B approaches the held object 80. Furthermore, the control unit 12 determines that the end effector model 30 cannot be projected when at least a part of the end effector model 30 overlaps with the obstacle region 44. In this manner, it is possible to prevent the end effector 2B from colliding with an obstacle when holding the holding object 80.

[0059] When the finger position 32 is not specified in the end effector model 30, the control unit 12 determines whether the end effector model 30 can be projected based on the feature points of the fingers associated with the end effector model 30.

[0060] It can also be said that the position of the end effector 2B corresponding to the position and rotation angle of the end effector model 30 that can be projected onto the surrounding environment map 40 is a position that fits the surrounding environment map 40. It can also be said that the control unit 12 determines the holding position from among the positions that fit the surrounding environment map 40.

[0061] The control unit 12 moves the approach position 70 so as to scan within the range of the window 22 of the mask image 20, and rotates the end effector model 30 at each position to project it onto the surrounding environment map 40. The control unit 12 extracts a combination of the approach position 70 and the rotation angle of the end effector model 30 that allows the end effector model 30 to be projected onto the surrounding environment map 40.

[0062] Furthermore, the control unit 12 calculates the degree of coincidence in the surrounding environment map 40 for each combination of the approach position 70 at which the end effector model 30 can be projected and the rotation angle of the end effector model 30. Specifically, the control unit 12 calculates the average value of the numerical value or color brightness set at each coordinate outside the range of the object region 42 within the range overlapping with the projection model 72a in the surrounding environment map 40 in Fig. 10 as the degree of coincidence of the projection model 72a. The control unit 12 calculates the average value of the numerical value or color brightness set at each coordinate outside the range of the object region 42 within the range overlapping with the projection model 72b in the surrounding environment map 40 in Fig. 10 as the degree of coincidence of the projection model 72b.

[0063] The control unit 12 also projects the extracted combination of the approach position 70 and the rotation angle of the end effector model 30 onto the object map 50. The control unit 12 projects the projection model 72a or 72b onto the object map 50, for example, as shown in FIG. 12A. The control unit 12 calculates the average value of the numerical value or color brightness value set at each coordinate of the range that overlaps with the window 22 of the mask image 20 in the object map 50 and overlaps with the projection model 72a in the cross section 52 of the held object 80, as the degree of agreement of the projection model 72a. The control unit 12 calculates the average value of the numerical value or color brightness value set at each coordinate of the range that overlaps with the window 22 of the mask image 20 in the object map 50 and overlaps with the projection model 72b in the cross section 52 of the held object 80, as the degree of agreement of the projection model 72b. In FIG. 12A, the solid black line surrounding the cross section 52 of the object map 50 is simply a line representing the contour of the cross section 52. The coordinates marked with a solid black line do not represent invalid holding positions.

[0064] The object map 50 represents the suitability of a position for holding the holding target 80. It can also be said that the position of the end effector 2B corresponding to the position and rotation angle of the end effector model 30 projected onto the object map 50 is a position that matches the object map 50. It can also be said that the control unit 12 determines the holding position from among the positions that match the object map 50.

[0065] The control unit 12 also projects the extracted combination of the approach position 70 and the rotation angle of the end effector model 30 onto the contact map 60. The control unit 12 projects the projection model 72a or 72b onto the contact map 60, for example, as shown in FIG. 12B. The control unit 12 calculates the average value of the numerical value or color brightness value set at each coordinate of the outer periphery 62 of the cross section of the holding object 80 in the contact map 60 that overlaps with the projection model 72a as the degree of agreement of the projection model 72a. The control unit 12 calculates the average value of the numerical value or color brightness value set at each coordinate of the outer periphery 62 of the cross section of the holding object 80 in the contact map 60 that overlaps with the projection model 72b as the degree of agreement of the projection model 72b. In FIG. 12B, the solid black line surrounding the outer periphery 62 of the contact map 60 is simply a line representing the outline of the outer periphery 62. It does not indicate that the coordinates on which the solid black line is drawn are inappropriate holding positions.

[0066] The contact map 60 represents the suitability of the contact portion of the end effector 2B with the holding object 80 as a position for contacting the holding object 80. It can also be said that the control unit 12 determines the holding position from among a plurality of positions of the end effector 2B corresponding to the positions and rotation angles of a plurality of end effector models 30 projected onto the contact map 60.

[0067] The control unit 12 may calculate the angle at which each of the two finger positions 32 of the end effector model 30 is incident on the outer periphery 62 along the direction of the stroke range 34. In other words, the control unit 12 may calculate the angle of intersection at each of two intersection points between the outer periphery 62 and a line along the direction of the stroke range 34 of the end effector model 30. In this embodiment, the control unit 12 calculates the incident angle as 0 degrees when the finger position 32 is perpendicularly incident on the outer periphery 62. The control unit 12 may reflect the calculated angle in the value of the degree of agreement in the contact map 60. The control unit 12 may calculate the degree of agreement as a larger value as the angle approaches 0 degrees. The control unit 12 may calculate, for example, the degree of agreement by the product of the average value of the brightness value of the numerical value or color set at each coordinate of the outer periphery 62 of the contact map 60 in the range overlapping with the projected end effector model 30 and the cosine value of the calculated angle.

[0068] When the outer periphery 62 has an uneven surface, the control unit 12 may generate a model in which the outer periphery 62 is flattened based on the thickness, width, or length of the finger of the end effector 2B, and calculate the angle at which the finger position 32 is incident on the flattened model.

[0069] The control unit 12 adds up the degrees of match calculated in each rule map for each combination of the approach position 70 and the rotation angle of the end effector model 30 to calculate an overall degree of match. The control unit 12 may weight the degrees of match calculated in each rule map and add them up. The control unit 12 weights the degrees of match calculated in each rule map the same for all combinations. The weighting coefficient applied to the degrees of match calculated in each rule map is also called a map coefficient. The map coefficient may be determined for each map.

[0070] As described above, the control unit 12 calculates the overall degree of agreement for each holding position based on the end effector model 30 and the rule map. The overall degree of agreement corresponds to an appropriateness value that indicates the appropriateness of each holding position. Furthermore, the rule map indicates the appropriateness of a position for holding the object to be held 80 by a numerical value or a color brightness value assigned to each position (each coordinate) of the rule map. The control unit 12 can calculate the appropriateness value by calculating the value assigned to each position when the end effector model 30 is superimposed on the rule map.

[0071] The control unit 12 compares the calculated overall coincidences for each combination of the approach position 70 and the rotation angle of the end effector model 30, and selects a combination with a high overall coincidence. The control unit 12 determines the position at which the finger of the end effector 2B enters the holding object 80 when it moves along the stroke direction, which is determined based on the approach position 70 and the rotation angle of the end effector model 30 of the selected combination, as the position at which the holding object 80 is held. That is, the control unit 12 determines the holding position based on the end effector model 30 and the rule map. The control unit 12 may determine the holding position by, for example, comparing the end effector model 30 with the rule map to estimate the holding position, and then adjusting or correcting the estimated holding position based on another rule. The control unit 12 may also determine the holding position based on an appropriate value. The control unit 12 outputs the determined holding position to the robot control device 110 via the interface 14.

[0072] (Example of procedure for determining holding position) The control unit 12 of the holding position determination device 10 may execute a holding position determination method including the procedure of the flowchart illustrated in Fig. 13. The holding position determination method may be realized as a holding position determination program executed by a processor constituting the control unit 12 of the holding position determination device 10. The holding position determination program may be stored in a non-transitory computer-readable medium.

[0073] The control unit 12 acquires data including an image of the holding object 80 and depth data of the holding object 80 (step S1). The control unit 12 generates an end effector model 30 (step S2). The control unit 12 determines the height of the position at which the holding object 80 is held (step S3). The control unit 12 generates a mask image 20 (step S4).

[0074] The control unit 12 generates rule maps such as the surrounding environment map 40, the object map 50, or the contact map 60 (step S5). The control unit 12 projects the end effector model 30 onto each rule map (step S6). The control unit 12 calculates the degree of agreement between each rule map and the end effector model 30 projected onto each rule map (step S7). The control unit 12 weights the degree of agreement calculated for each rule map in order to calculate the overall degree of agreement (step S8). The control unit 12 selects a projection position of the end effector model 30 when the overall degree of agreement is high, and determines a position where the finger of the end effector 2B enters the holding object 80 at the selected position as the holding position (step S9). After executing the procedure of step S9, the control unit 12 ends the execution of the procedure of the flowchart in FIG. 13.

[0075] The control unit 12 may execute the procedure of step S4 for generating the mask image 20 before step S2 or S3.

[0076] (Small summary) As described above, according to the holding position determination device 10 and the holding position determination method of the present embodiment, the holding position when the holding object 80 is held by the end effector 2B of the robot 2 is determined based on the rule map. By determining the holding position based on the rule map, the experience of the worker can be reflected in the rule map. For example, when the worker holds various holding objects 80, the worker considers which position to hold each of the holding objects 80. The worker decides the holding position by considering, for example, the center of gravity of the holding object 80, obstacles existing around the holding object 80, or the wide range of positions to hold the holding object 80. In other words, according to the holding position determination device 10 and the holding position determination method of the present embodiment, the thinking of the worker is reflected in the rule map, the degree of agreement in each rule map is calculated, and the gripping position is determined based on the overall degree of agreement calculated by weighting the degree of agreement. As a result, the robot 2 can hold the holding object 80 at the holding position considered by the worker. In other words, the holding position of the object can be easily determined at a position according to the human intention.

[0077] Also, a worker can use a rule map that defines the holding position as a rule. As a result, learning is not required. Also, by adding a rule map when a new rule is generated, it becomes easier to respond to changes in the environment. For example, a task that includes holding an object to be held 80 may include various components according to the worker or the environment of the work site. Therefore, it is necessary to determine the holding position of the object to be held 80 based on each component and hold it. By generating a rule map that reflects various components and adding them to the objects to be calculated for the degree of coincidence, it becomes easier to respond to special rules of each worker or each site.

[0078] (Other embodiments) Other embodiments are described below.

[0079] <Use of estimation results based on other methods> The holding position determination device 10 may acquire the holding position of the holding object 80 estimated based on another method. The holding position determination device 10 may determine the holding position from among the acquired holding positions by executing the holding position determination method according to the present embodiment on the acquired holding positions.

[0080] Specifically, the control unit 12 of the holding position determination device 10 acquires the holding position of the holding target 80 estimated based on another technique via the interface 14. The control unit 12 calculates a combination of the approach position 70 and the rotation angle of the end effector model 30 corresponding to the acquired holding position. The control unit 12 projects the end effector model 30 onto each rule map with the combination corresponding to the acquired holding position, and calculates the degree of agreement in each rule map. The control unit 12 weights and sums up the degrees of agreement in each rule map to calculate an overall degree of agreement. The control unit 12 selects the holding position corresponding to the combination with the largest value of the overall degree of agreement, and outputs it to the robot control device 110.

[0081] The holding position determination device 10 may calculate a total degree of agreement for the acquired holding position to evaluate the appropriateness of the acquired holding position. The control unit 12 evaluates whether the acquired holding position is appropriate based on the calculated total degree of agreement. For example, the control unit 12 may determine that the acquired holding position is appropriate when the calculated total degree of agreement is equal to or greater than a predetermined value.

[0082] The holding position determination device 10 may acquire the center position and rotation angle of the end effector 2B estimated based on another method. The holding position determination device 10 can calculate a combination of the approach position 70 and the rotation angle by regarding the acquired center position of the end effector 2B as the approach position 70, and execute the holding position determination method according to this embodiment.

[0083] The holding position determination device 10 can reduce the number of combinations of the approach position 70 and the rotation angle to be used for calculating the degree of coincidence by acquiring the holding position of the holding object 80 estimated based on another method, thereby reducing the calculation load.

[0084] <Adjusting weighting through learning> The holding position determination device 10 calculates the overall degree of coincidence by weighting and summing the degrees of coincidence calculated for each rule map. The holding position determination device 10 may update the weighting coefficient by learning based on annotation information for the determined holding position. The holding position determination device 10 can improve the accuracy of determining the holding position by updating the weighting coefficient.

[0085] Specifically, the control unit 12 of the holding position determination device 10 may notify the user of the determined holding position via the interface 14. The control unit 12 may receive an input from the user to correct the holding position as an annotation for the determined holding position via the interface 14. The control unit 12 may output the corrected holding position to the robot control device 110 based on the information of the correction by the user. The control unit 12 may update the weighting coefficient by learning based on the information of the correction by the user, and may redo the determination of the holding position.

[0086] The control unit 12 may determine a plurality of holding position candidates and notify the user through the interface 14. The control unit 12 may determine a holding position having an overall degree of coincidence equal to or greater than a predetermined value as a candidate. The control unit 12 may receive an input from the user through the interface 14 to select from the holding position candidates as an annotation for the determined holding position. The control unit 12 may output the holding position selected by the user to the robot control device 110. The control unit 12 may update the weighting coefficient by learning based on the information selected by the user.

[0087] The control unit 12 may extract a holding position where the overall degree of agreement is equal to or greater than a predetermined value as a candidate position where the holding object 80 can be held, and notify the user. The control unit 12 receives an input for correcting the candidate position or an input for selecting the candidate position from the user through the interface 14 as an annotation for the candidate position. The control unit 12 evaluates the suitability of each candidate position based on the input for correcting or selecting the candidate position. The control unit 12 may update the weighting coefficient so that the value of the overall degree of agreement is large for a candidate position that is evaluated to have a high suitability as a holding position based on the user's input. The control unit 12 may output the selected candidate position to the robot control device 110 as the holding position. The control unit 12 may output the corrected candidate position to the robot control device 110 as the holding position.

[0088] The control unit 12 of the holding position determination device 10 may execute the procedure of the flowchart shown in Fig. 14. The control unit 12 selects candidates for the holding position by executing the procedure of the flowchart of Fig. 13, for example (step S11). The control unit 12 outputs the candidates for the holding position to the interface 14 (step S12). The control unit 12 corrects the holding position based on a correction input by the user (step S13). The control unit 12 learns the correction content (step S14). The control unit 12 updates the weighting based on the learning result (step S15). After executing the procedure of step S15, the control unit 12 ends the execution of the procedure of the flowchart of Fig. 14.

[0089] As described above, the holding position determination device 10 can update the weighting based on the contents of the annotation by the user. As a result, the robustness capable of adapting to various working environments can be improved.

[0090] The holding position determination device 10 performs weighting on and mixes multiple maps to generate a rule map. For example, the holding position determination device 10 mixes multiple object maps 50 to generate one object map 50. When performing weighting on and mixes multiple maps, the holding position determination device 10 may update the weighting coefficient for each map by learning based on annotations.

[0091] The holding position determination device 10 may obtain annotation information for a plurality of holding positions, and correct the appropriate value based on the annotation information.

[0092] <Image-based map generation> The holding position determination device 10 may generate a rule map based on an image captured by the camera 4. The control unit 12 of the holding position determination device 10 may, for example, estimate the center of gravity of the holding object 80 based on the image and generate an object map 50 that specifies the center of gravity position. The control unit 12 may, for example, estimate the material of the holding object 80 based on the image and generate a contact map 60 that specifies the friction force acting between the surface of the holding object 80 and the fingers of the end effector 2B. The control unit 12 may estimate the material of the holding object 80 based on the color or pattern of the holding object 80, or information regarding unevenness.

[0093] <Setting the height of the holding position> The holding position determination device 10 first sets the height of the position at which the holding object 80 is held, and determines the holding position at the set height. The holding position determination device 10 may change the height of the position at which the holding object 80 is held, execute the holding position determination method at each height, and determine the holding height, approach position 70 of the end effector 2B, and rotation angle combination at which the overall coincidence value is maximized as the holding position. In this way, the stability of holding can be improved.

[0094] The above describes an embodiment of the holding position determination device 10. However, the embodiment of the present disclosure can also be implemented as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0095] The implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be configured so that a part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as necessary.

[0096] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions, etc. included in each component, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, etc. can be combined into one or divided.

[0097] All of the features described in this disclosure and / or all steps of any method or process disclosed herein may be combined in any combination except those combinations in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.

[0098] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.

[0099] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure may have their numbers exchanged. For example, the first region may exchange identifiers "first" and "second" with the second region. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a code. Descriptions of identifiers such as "first" and "second" in this disclosure alone should not be used to interpret the order of the configurations or to justify the existence of identifiers with smaller numbers. [Explanation of symbols]

[0100] 10 Holding position determination device (12: control unit, 14: interface) 20 mask images (22: window, 24: mask) 30 End effector model (32: finger position, 34: stroke range, 36: finger motion range, 38: non-motion range) 40 Surrounding environment map (42: object area, 44: obstacle area, 46: approach area, 46a: boundary, 48: non-approach area) 50 Object Map (52: Cross Section) 60 Contact Map (62: Outer perimeter) 70 Approach position 72a, 72b, 72c Projection Models 74a, 74c Projection position of fingers 80 Holding object (82: Holding point) 100 Robot control system (2: Robot, 2A: Arm, 2B: End effector, 4: Camera, 5: Robot motion range, 6: Work start table, 7: Work target table, 110: Robot control device)

Claims

1. A control unit is provided that determines a position where an end effector having two or more fingers that holds a holding object comes into contact with the holding object as a holding position, The control unit is obtaining an end effector model including a stroke model that indicates a range in which the finger moves to open and close; acquiring a rule map including a map that defines a position of the object to be held by the end effector based on a held object image obtained by photographing the object to be held; determining the holding position based on the end effector model and the rule map; Holding positioning device.

2. The holding position determination device according to claim 1 , wherein the control unit acquires the rule map based on at least one of shape data of the object to be held and depth data associated with the image to be held.

3. The holding position determination device according to claim 1 , wherein the rule map includes a map generated based on a position in a height direction of the holding object that holds the holding object.

4. The control unit is calculating a suitability value representing suitability of the holding position for each of the plurality of holding positions based on the end effector model and the rule map; The holding position determination device according to claim 1 , wherein the holding position is determined based on the appropriate value.

5. the end effector model and the rule map are expressed by values ​​indicating suitability of each position for holding the object to be held, The holding position determining device according to claim 4 , wherein the control unit calculates the appropriate value by calculating a value assigned to each position when the end effector model is superimposed on the rule map.

6. The holding position determination device according to claim 4 , wherein the control unit calculates the appropriate value based on the rule map based on a plurality of the maps included in the rule map and a map coefficient defined for each of the maps.

7. The control unit is Obtaining annotation information for the plurality of holding locations; The holding position determining device according to claim 4 , wherein the appropriate value is corrected based on information of the annotation.

8. the rule map includes a surrounding environment map that defines a range in which the end effector can move when the end effector holds the holding object, The holding position determination device according to claim 1 , wherein the control unit determines the holding position from a plurality of positions of the end effector that match the surrounding environment map.

9. A first area in which the end effector may exist and a second area in which the end effector may not exist are defined in the surrounding environment map; 9. The holding position determination device according to claim 8, wherein the second area is defined at least in the holding object and an area surrounding the holding object, and in the surrounding area, the second area is allocated such that the proportion of the second area in the first area decreases as the distance from the holding object increases.

10. the rule map includes an object map which is a map representing suitability of a position for holding the holding object, The holding position determination device according to claim 1 , wherein the control unit determines the holding position from a plurality of positions of the end effector that match the object map.

11. The holding position determination device according to claim 10 , wherein the object map is defined based on characteristics of the object to be held.

12. 12. The holding position determination device according to claim 11, wherein the object map is defined based on any one of the shape, material, texture, weight, and friction coefficient of the object to be held.

13. the rule map includes a contact map which is a map representing suitability of a contact portion of the end effector with the held object as a position for contacting the held object, The holding position determination device according to claim 1 , wherein the control unit determines the holding position from a plurality of positions of the end effector relative to the contact map.

14. The holding position determination device according to claim 13 , wherein the contact map is defined based on a shape of a contact portion of the end effector with the object to be held and a shape of the object to be held.

15. The end effector is a gripper that holds the holding object by pinching it between at least two fingers, the end effector model defines a range of motion for the finger; The holding position determination device according to claim 1 , wherein the control unit determines, as the holding position, a combination of positions at which the fingers come into contact with the holding object when the gripper grasps the holding object.

16. the end effector has a suction portion that holds the object to be held by suction, the end effector model defines a range in which the suction unit interferes with another object when suctioning the object to be held, The holding position determination device according to claim 1 , wherein the control unit determines, as the holding position, a position at which the suction unit comes into contact with the holding object when the end effector suctions the holding object.

17. A holding position determination device that determines a position where an end effector having two or more fingers that holds a holding object comes into contact with the holding object as a holding position acquires an end effector model including a stroke model that indicates a range in which the fingers open and close; The holding position determination device acquires a rule map including a map that specifies a position of the holding object to be held by the end effector based on a holding object image obtained by photographing the holding object; the holding position determination device determines the holding position based on the end effector model and the rule map; A holding position determination method comprising:

Images