Picking system, control device, picking method, program, and storage medium

Abstract

To provide a picking system, a control device, a picking method, a program, and a storage medium that can reduce the time necessary for the picking task.SOLUTION: According to an embodiment, a picking system includes a picking robot and a control device. The picking robot transfers an object from a first space to a second space by using a robot hand. The control device controls the picking robot. When a first measurement result related to a shape of the object in the first space in a view along a first direction, is acquired, the control device performs a first calculation of calculating a position candidate for placing the object in the second space based on the first measurement result. Furthermore, when a second measurement result related to a shape of the object in a view along a second direction crossing the first direction, is acquired during action of the robot hand onto the object, the control device performs a second calculation of calculating a position of the robot hand as of when placing the object in the second space, on the basis of the second measurement result and the position candidate.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a picking system, a control device, a picking method, a program, and a storage medium. [Background technology]

[0002] There are picking systems that transfer objects, and there is a demand for technology that can reduce the time required for picking operations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-54660 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a picking system, a control device, a picking method, a program, and a storage medium that can reduce the time required for picking work. [Means for solving the problem]

[0005] A picking system according to an embodiment includes a picking robot and a control device. The picking robot transfers an object from a first space to a second space using a robot hand. The control device controls the picking robot. When the control device acquires a first measurement result regarding the shape of the object in the first space in a first direction, the control device performs a first calculation to calculate a position candidate when placing the object in the second space based on the first measurement result. Furthermore, when the control device acquires a second measurement result regarding the shape of the object in a second direction intersecting the first direction while the robot hand is acting on the object, the control device performs a second calculation to calculate a position of the robot hand when placing the object in the second space based on the second measurement result and the position candidate. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing a picking system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of the control device. [Figure 3] 3(a) to 3(c) are schematic diagrams for explaining a measurement method using the second measuring device. [Figure 4] FIG. 4 is a schematic diagram showing a processing procedure performed by the picking system according to the embodiment. [Figure 5] FIG. 5 is a flowchart showing the processing performed by the placement plan generating unit. [Figure 6] FIG. 6 is a flowchart showing the calculation process performed by the position candidate calculation unit. [Figure 7] FIG. 7 is a flowchart showing a method for searching for location candidates by the location candidate calculation unit. [Figure 8] FIG. 8 is a flowchart showing an outline of the processing by the hand position calculation unit. [Figure 9] FIG. 9 is a flowchart showing the processing by the hand position calculation unit. [Figure 10] FIG. 10 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In the present specification and the drawings, elements similar to those already explained are given the same reference numerals and detailed explanations will be omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing a picking system according to an embodiment. As shown in FIG. 1, a picking system 1 according to the embodiment includes a picking robot 10, a first measuring instrument 21, a second measuring instrument 22, a third measuring instrument 23, and a control device 30.

[0009] Here, for the purpose of describing the embodiment, an X direction, a Y direction (second direction), and a Z direction (first direction) are used. The X direction and the Y direction intersect with each other. The Z direction intersects with the XY plane (first plane). For example, the X direction and the Y direction are parallel to the horizontal plane. The Z direction is parallel to the vertical direction.

[0010] The picking robot 10 transfers an object placed in a first space SP1 in a first container 41 to a second space SP2 in a second container 42. More specifically, the first container 41 has a first opening OP1 facing the Z direction. The second container 42 has a second opening OP2 facing the Z direction. The picking robot 10 takes the object out of the first container 41 through the first opening OP1 and carries the object into the second container 42 through the second opening OP2. The picking robot 10 includes a robot hand 11, a robot arm 12, and a housing 13.

[0011] The robot hand 11 holds (stably grasps) an object. For example, the robot hand 11 holds the object by suction, pinching, or jamming. In the example of FIG. 1, the robot hand 11 includes multiple fingers 11a. The object is held by pinching the object between the multiple fingers 11a. The robot hand 11 is attached to a robot arm 12.

[0012] The robot arm 12 moves the robot hand 11. In the example shown in FIG. 1 , the robot arm 12 is a vertical articulated robot having six degrees of freedom, and the robot hand 11 is attached to the tip of the robot arm 12. The robot arm 12 may be a horizontal articulated robot, a linear motion robot, a Cartesian robot, or a parallel link robot. The robot arm 12 may include a combination of two or more selected from the vertical articulated robot, the horizontal articulated robot, the linear motion robot, the Cartesian robot, and the parallel link robot. The robot arm 12 is attached to a housing 13.

[0013] The housing 13 supports the robot arm 12 and is fixed to the floor. The housing 13 may also house a power supply unit for driving an electric actuator such as a motor, a cylinder, a tank, and a compressor for driving a fluid actuator, various safety mechanisms, etc. The control device 30 may also be housed in the housing 13.

[0014] The first measuring instrument 21 measures the shape of an object placed in the first space SP1 in the Z direction. For example, the first measuring instrument 21 includes an imaging unit 21a. The imaging unit 21a is a camera including one or two selected from an image sensor and a distance measurement sensor. The imaging unit 21a captures an image of the object in the first space SP1 in the Z direction and acquires an image (still image). The imaging unit 21a may also acquire a moving image, and a still image may be extracted from the moving image. The imaging unit 21a transmits the image to the control device 30.

[0015] The control device 30 measures the shape of a first surface (top surface) of the object that intersects with the Z direction from the image. In the picking system 1, the photographing unit 21a and the control device 30 function as the first measuring device 21. The measurement result (first measurement result) by the first measuring device 21 includes first shape information regarding the shape of the first surface of each object. An image processing device separate from the control device 30 may be incorporated into the photographing unit 21a and used as the first measuring device 21.

[0016] The second measuring instrument 22 measures the shape of an object in the Y direction while being acted upon by the picking robot 10. For example, the second measuring instrument 22 includes a light curtain 22a. The light curtain 22a includes a light-projecting unit 22a1 and a light-receiving unit 22a2. The light curtain 22a has a sensing area SR that faces the first opening OP1 in the Z direction. The sensing area SR is an area through which light emitted from the light-projecting unit 22a1 passes. The light curtain 22a detects the passage of an object through the sensing area SR. The light curtain 22a transmits the detection result to the control device 30.

[0017] The control device 30 measures the shape of the second surface (side surface) of the object that intersects with the Y direction from the position of the light curtain 22a in the Z direction, the time when the object passes through the sensing region SR, the position of the robot hand 11 in the Z direction at that time, etc. In the picking system 1, the light curtain 22a and the control device 30 function as the second measuring device 22. The measurement result by the second measuring device 22 (second measurement result) includes second shape information regarding the shape of the second surface. A calculation device separate from the control device 30 may be provided together with the light curtain 22a and used as the second measuring device 22.

[0018] The second measuring instrument 22 may include a distance measuring sensor such as a laser range finder or an imaging unit instead of the light curtain 22a. The control device 30 measures the length of the object in the Z direction based on the measurement results or images from the distance measuring sensor.

[0019] The third measuring instrument 23 measures the shape of an object placed in the second space SP2 in the Z direction. For example, the third measuring instrument 23 includes an imaging unit 23a. The imaging unit 23a is a camera including one or two selected from an image sensor and a distance measurement sensor. The imaging unit 23a images the second space SP2 in the Z direction and acquires an image (still image). The imaging unit 23a may also acquire a moving image, and a still image may be extracted from the moving image. The imaging unit 23a transmits the image to the control device 30.

[0020] The control device 30 measures the shape of the object placed in the second space SP2 from the image. In the picking system 1, the photographing unit 23a and the control device 30 function as the third measuring device 23. The measurement result by the third measuring device 23 (third measurement result) includes obstacle information related to the three-dimensional shape of the object placed in the second space SP2. An image processing device separate from the control device 30 may be incorporated into the photographing unit 23a and used as the third measuring device 23.

[0021] In addition to the calculations described above, the control device 30 also controls the picking robot 10. For example, the control device 30 operates each drive axis of the robot arm 12 to move the robot hand 11 and adjust the posture of the robot hand 11. The control device 30 also causes the robot hand 11 to hold an object and release the object.

[0022] FIG. 2 is a block diagram illustrating a functional configuration of the control device. The control device 30 includes an integration unit 31, a measurement information processing unit 32, a holding plan generating unit 33, a placement plan generating unit , a motion plan generating unit 35, and a robot control device .

[0023] Integration Division 31 、 The picking system 1 generates, operates, and manages work plans based on information input by the user from the external interface (I / F) 37, picking instructions input from a higher-level system, the status of the picking system 1, and the like.

[0024] The measurement information processing unit 32 controls the photographing unit 21a, the light curtain 22a, and the photographing unit 23a. The measurement information processing unit 32 processes information obtained from the photographing unit 21a, the light curtain 22a, and the photographing unit 23a, and generates information necessary for operation planning, operation control, error detection, etc. The measurement information processing unit 32 performs some of the functions of the first measuring instrument 21 to the third measuring instrument 23.

[0025] For example, the measurement information processing unit 32 segments the image captured by the image capturing unit 21a and generates first shape information using the segmentation results. During segmentation, each item in the image is identified and the image is divided into one or more regions. Each region corresponds to an object. The first shape information relates to the shape of the first surface of each object in the first container 41, and includes the segmentation results of the first surface, the length of each object's first surface in the X direction and the length of each object's first surface in the Y direction, and the position of the object's first surface on the XY plane. The length and position are calculated based on the segmentation results. For example, the actual length of the object is calculated based on the distance between the image capturing unit 21a and each object and the length (number of pixels) of the object in the X direction or Y direction in the image. Similarly, the position of the object's first surface on the XY plane is calculated based on the distance between the image capturing unit 21a and each object and the position of the first surface in the image.

[0026] The measurement information processing unit 32 generates second shape information from the detection result by the light curtain 22a. The second shape information relates to the shape of at least a part of the second surface of the held object. Specifically, the second shape information includes the length (height) of the at least part of the second surface in the Z direction.

[0027] 3(a) to 3(c) are schematic diagrams for explaining a measurement method using the second measuring device. A method for measuring the length of an object in the Z direction will be described with reference to Figures 3(a) and 3(b). The control device 30 records the angle of each joint of the robot arm 12 at predetermined intervals. The light curtain 22a also detects the presence or absence of an obstruction between the light-projecting unit 22a1 and the light-receiving unit 22a2 at predetermined intervals.

[0028] As shown in Figure 3(a), when the picking robot 10 holds an object, at least a part of the light L emitted from the light-emitting unit 22a1 is blocked by the robot hand 11 or the robot arm 12 and does not enter the light-receiving unit 22a2. When the picking robot 10 lifts the object, as shown in Figures 3(b) and 3(c), when one end (lower end) of the object in the Z direction passes through the sensing region SR, the light L enters the light-receiving unit 22a2.

[0029] The light curtain 22a records a second time t2, which is the first time the light L is detected by the light receiving unit 22a2 after it has been blocked. The control device 30 calculates a first position z1 of the robot hand 11 in the Z direction at the first time from the angles of the joints of the robot arm 12 at a first time t1 immediately before the second time t2. For example, the position of the robot hand 11 is calculated as the position of the tool center point (TCP) of the robot hand 11. The control device 30 calculates a second position z2 of the robot hand 11 in the Z direction at the second time from the angles of the joints of the robot arm 12 at the second time t2. The control device 30 estimates (z2 + z1) / 2 as the position zH of the robot hand 11 when the object passes through the sensing region SR of the light curtain 22a. The control device 30 references the position zL of the light curtain 22a in the Z direction. The position zL of the light curtain 22a is registered in advance. The control device 30 calculates zH-zL as the height SZ of the object.

[0030] As shown in FIG. 3(c), the range measured by the second measuring instrument 22 does not have to cover the entire second surface of the object. The second measuring instrument 22 is only required to measure the length (distance) between the TCP and the bottom end of the object. In other words, this length is the amount of protrusion of the object from the tip of the robot hand 11. This length may be calculated based on a reference other than the TCP. For example, if a control point is set at the tip of the robot arm 12, the length is calculated based on this control point. If the control point is a point mechanically fixed to a part of the robot arm 12, the length can be measured based on that point. Furthermore, depending on the configuration of the robot hand 11, the amount of protrusion measured by the second measuring instrument 22 may be equal to the actual length of the object in the Z direction. For example, if the robot hand 11 holds only the top surface of the object by suction, the length of the entire object in the Z direction may be calculated as the amount of protrusion.

[0031] The measurement information processing unit 32 generates obstacle information from the image captured by the imaging unit 23a. The obstacle information includes the position of each object in the XY plane in the second space SP2, the position of the top surface of each object in the Z direction, etc.

[0032] The holding plan generating unit 33 generates a holding plan. The holding plan includes an object holding method, a holding position of the robot arm 12 when holding the object, a holding posture, and waypoints to reach the holding position.

[0033] The placement plan generator 34 generates a placement plan. The placement plan includes the placement position and placement posture of the robot arm 12 when releasing the held object in the second container 42, and waypoints along the way to reach the placement position.

[0034] The motion plan generation unit 35 generates motion information of the robot arm 12. The motion information includes information on a holding motion, a transport motion, and a placement motion. The holding motion is a motion in which the tip of the robot arm 12 moves from above the holding position to the holding position and holding posture. The transport motion is a motion in which the tip of the robot arm 12 moves from above the holding position to above the placement position. The placement motion is a motion in which the tip of the robot arm 12 moves from above the placement position to the placement position and placement posture.

[0035] The robot control device 36 controls the picking system 1 including the picking robot 10 in accordance with information generated by the holding plan generation unit 33, the placement plan generation unit 34, or the operation plan generation unit 35, and instructions for switching between operations from the integration unit 31.

[0036] The external I / F 37 executes input and output of data between the integration unit 31 (control device 30) and an external device (not shown).

[0037] FIG. 4 is a schematic diagram showing a processing procedure performed by the picking system according to the embodiment. The integration unit 31 receives a picking instruction from the external I / F 37 (step S0). The picking instruction is transmitted, for example, from a higher-level host computer. The integration unit 31 instructs the measurement information processing unit 32 to photograph the first container 41. The measurement information processing unit 32 causes the photographing unit 21a to photograph the interior of the first container 41 (step S1) and generates first shape information. After photographing the first container 41, the holding plan generation unit 33 generates a holding plan (step S2). In parallel, the measurement information processing unit 32 causes the photographing unit 23a to photograph the interior of the second container 42 (step S3) and generates obstacle information.

[0038] After the holding plan generation unit 33 has completed generation of the holding plan, the robot control device 36 executes a holding operation based on the generated holding plan (step S4). In parallel, the placement plan generation unit 34 calculates position candidates for when the transferred object is placed in the second space SP2 based on the holding plan and the imaging results of the second container 42 (step S5). The placement plan generation unit 34 calculates the priority of the position candidates (step S6). The placement plan generation unit 34 saves the calculated position candidates and priorities. After completing the holding operation, the robot control device 36 executes a transfer operation (step S7). In the transfer operation, the held object is lifted and transferred to the second container 42. The measurement information processing unit 32 causes the light curtain 22a to detect the held object during the transfer operation (step S8) and generates second shape information.

[0039] The placement plan generator 34 calculates the position of the robot hand 11 when placing the object in the second container 42 based on the second shape information and the position candidates (step S9). Hereinafter, the position of the robot hand 11 when placing the object in the second container 42 will be referred to as the "hand position." After calculating the hand position, the robot control device 36 executes the placement operation (step S10). After the placement operation is completed, it is determined whether the specified number of objects has been transferred (step S11). Steps S1 to S10 are repeated until the specified number of objects has been transferred.

[0040] FIG. 5 is a flowchart showing the processing performed by the placement plan generating unit. The placement plan generation unit 34 includes a position candidate calculation unit 34a and a hand position calculation unit 34b. When generation of the holding plan is completed, the position candidate calculation unit 34a starts processing. The position candidate calculation unit 34a calculates position candidates of the object in the second container 42 based on the first shape information, obstacle information, and the holding plan (step S5). The position candidates are candidates for the position when the object to be transferred is placed in the second container 42. Next, the position candidate calculation unit 34a calculates the priority of each position candidate (step S6). The position candidate calculation unit 34a stores the position candidates and the priority inside the placement plan generation unit 34.

[0041] The hand position calculation unit 34b starts processing when the measurement of the object by the second measuring instrument 22 is completed. The hand position calculation unit 34b calculates the hand position of the object using the position candidate calculated by the position candidate calculation unit 34a and the second shape information obtained by the second measuring instrument 22 (step S9). In addition, when calculating the hand position, the robot corresponding to the hand position is Arm 12 Then, the motion plan generating unit 35 calculates the position of the robot. Arm 12 Based on the position of the object, action information is generated, and a placement action is performed based on the action information.

[0042] FIG. 6 is a flowchart showing the calculation process performed by the position candidate calculation unit. With reference to FIG. 6, the calculation process (first calculation) by the position candidate calculation unit 34a for calculating position candidates will be described. First, first shape information of the object to be transferred is acquired (step S51). More specifically, the size in the XY plane and the segmentation result included in the first shape information are acquired. Planar mesh data is generated using the segmentation result (step S52). The mesh data is generated by dividing a part of the image divided by segmentation into a grid. The generated planar mesh data is saved as "MESH_OBJ". The planar mesh data indicates the shape in the XY plane of the first surface of the object to be transferred.

[0043] Obstacle information is acquired (step S53). Using the obstacle information, three-dimensional mesh data indicating the shape of the obstacle in the second space SP2 is generated (step S54). The generated three-dimensional mesh data is saved as "MESH_TOTE."

[0044] Using the planar mesh data and the three-dimensional mesh data, candidate positions for the placement of the held object are searched for (step S56). Examples of search methods include grid search, binary search tree, or Monte Carlo Tree Search (MCTS). Unless many objects are placed in the second container 42 or the second container 42 is excessively small, multiple candidate positions are usually obtained. Preferably, all possible positions are calculated as candidate positions. To reduce the calculation time for candidate positions, the number of candidate positions to be calculated may be specified in advance. In this case, the candidate position calculation unit 34a terminates the search when the specified number of candidate positions have been calculated.

[0045] The priority of each location candidate is calculated (step S6). The location candidate calculation unit 34a stores the location candidates and the priorities.

[0046] FIG. 7 is a flowchart showing a method for searching for location candidates by the location candidate calculation unit. In FIG. 7, X0 indicates the origin coordinate of the second container 42 in the X direction. Y0 indicates the origin coordinate of the second container 42 in the Y direction. Z0 indicates the origin coordinate of the second container 42 in the Z direction. For example, the position of the bottom surface of the second container 42 in the Z direction is set as Z0. For example, one of the four corners of the second container 42 is set as the origin position on the XY plane. The bottom surface of the second container 42 is set as the origin position in the Z direction. SX is the length of the second container 42 in the X direction. SY is the length of the second container 42 in the Y direction. SZ is the length of the second container 42 in the Z direction. SX, SY, and SZ are set in advance.

[0047] First, X0 is substituted for variable X (step S56a). It is determined whether variable X exceeds X0+SX (step S56b). That is, it is determined whether the X coordinate to be searched for is located outside the second container 42. If variable X exceeds X0+SX, the search ends. If variable X does not exceed X0+SX, Y0 is substituted for variable Y (step S56c). It is determined whether variable Y exceeds Y0+SY (step S56d). That is, it is determined whether the Y coordinate to be searched for is located outside the second container 42. If variable Y exceeds Y0+SY, a value obtained by adding ΔX to the current variable X is substituted for variable X (step S56e). Step S56b is executed again. That is, the X coordinate to be searched for is shifted slightly in the X direction.

[0048] In step S56d, if the variable Y does not exceed Y0+SY, Z0+SZ is assigned to the variable Z (step S56f). The variables X, Y, and Z at the time of completion of step S56f are set as the coordinates (X, Y, Z) of MESH_OBJ (step S56g). MESH_OBJ is planar mesh data of the object to be transferred. It is determined whether the coordinates (X, Y, Z) of MESH_OBJ intersect with MESH_TOTE (step S56h). MESH_TOTE is three-dimensional mesh data within the second container 42. In step S56h, it is determined whether the bottom surface of the object will come into contact with an obstacle (another object or the bottom or side surface of the second container 42) when the object is placed at the coordinates (X, Y, Z).

[0049] If the coordinates (X, Y, Z) do not intersect with MESH_TOTE, a value obtained by subtracting ΔZ from the current variable Z is assigned to variable Z (step S56i). Step S56g is executed again. That is, the position of the object to be placed is repeatedly lowered in the Z direction until the bottom surface of the object comes into contact with an obstacle. If the coordinates (X, Y, Z) intersect with MESH_TOTE, the coordinates (X, Y, Z) are saved as a position candidate (step S56j). The priority of the saved position candidate is calculated (step S6). Once the priority is calculated, a value obtained by adding ΔY to the current variable Y is assigned to variable Y (step S56k). Then, step S56d is executed again.

[0050] The method of calculating the priority can be set arbitrarily depending on the placement criteria to be emphasized. As an example, the object is preferentially placed from the corner or the bottom surface of the second container 42. In this case, the score Sc indicating the priority is calculated by the following mathematical formula 1. In this formula, a, b, and c are weighting coefficients. X, Y, and Z are the coordinates of the position candidate in the X direction, Y direction, and Z direction, respectively. X0 and Y0 are the coordinates on the XY plane of the corner where the object is to be preferentially placed. X1 and Y1 are the coordinates on the XY plane of the corner diagonally opposite the corner at the coordinates (X0, Y0). The larger the score Sc, the higher the priority.

number

[0051] 7, an example has been described in which a priority is calculated each time a location candidate is calculated. However, after multiple location candidates are calculated, a priority may be calculated for each location candidate.

[0052] FIG. 8 is a flowchart showing an outline of the processing by the hand position calculation unit. 8, the calculation process (second calculation) by the hand position calculation unit 34b for calculating the hand position will be described. First, second shape information is acquired, and the protrusion amount of the object in the Z direction (SZ_OBJ) is set (step S90). Three-dimensional mesh data "MESH_TOTE" indicating obstacle information in the second container 42 is acquired (step S91). As MESH_TOTE, the three-dimensional mesh data generated by the position candidate calculation unit 34a may be used, or new three-dimensional mesh data may be generated.

[0053] The actual holding position of the object by the robot hand 11 is acquired from the holding result (step S92). The shape of the hand is acquired as mesh data "MESH_HAND" (step S93). The acquired mesh data is arranged based on the actual holding position of the object by the robot hand 11. For example, the mesh data "MESH_HAND" is prepared in advance. The mesh data "MESH_HAND" may be generated based on an image acquired by the photographing unit 21a. The hand position is determined using the position candidate calculated by the position candidate calculation unit 34a, SZ_OBJ, the three-dimensional mesh data "MESH_TOTE", and the mesh data "MESH_HAND" (step S94).

[0054] FIG. 9 is a flowchart showing the processing by the hand position calculation unit. With reference to Fig. 9, step S94 in the flowchart shown in Fig. 8 will be described in detail. First, one position candidate is extracted from one or more position candidates calculated by the position candidate calculation unit 34a (step S94a). In step S94a, the position candidates are extracted in descending order of priority. The extracted position candidate is set as the coordinates (X_OBJ, Y_OBJ, and Z_OBJ) of the bottom surface of the object to be transferred (step S94b).

[0055] An example of a specific method for setting the coordinates (X_OBJ, Y_OBJ, and Z_OBJ) will be described. The actual holding position of the robot hand 11 in the XY plane acquired in step S92 is set to (X_GTCP, Y_GTCP). The coordinates of the object in the first container 41 are set to (X_GOBJ, Y_GOBJ). In this case, the relative position between the holding position of the robot hand 11 and the position of the object is (X_REL, Y_REL) = (X_GOBJ - X_GTCP, Y_GOBJ - Y_GTCP). When the position candidate of the object is (X_C, Y_C, Z_C), taking the relative position into consideration, Delivery The hand position candidates at the time of placement are (X_OBJ, Y_OBJ, Z_OBJ)=(X_C-X_REL, Y_C-Y_REL, Z_C).

[0056] The value obtained by adding the Z coordinate Z_OBJ of the bottom surface and the protrusion amount SZ_OBJ of the object in the Z direction is set as "Z_TCP" (step S94c). Z_TCP indicates the position of the TCP of the robot hand 11 in the Z direction. When MESH_HAND is placed on the coordinates (X_OBJ, Y_OBJ, and Z_TCP), it is determined whether MESH_HAND intersects with MESH_TOTE (step S94d).

[0057] If MESH_HAND intersects with MESH_TOTE, the value obtained by adding ΔZ to Z_TCP is set as the new Z_TCP (step S94e). It is then determined whether the amount of increase in the new Z_TCP relative to the Z_TCP set in step S94c exceeds a threshold (step S94f). The more ΔZ is added, the higher the object will be dropped from. The threshold is set to a height at which the object will not be damaged if dropped. If the amount of increase does not exceed the threshold, step S94d is executed again using the new Z_TCP.

[0058] In step S94d, if MESH_HAND does not intersect with MESH_TOTE, the angles of each joint of the robot arm 12 corresponding to Z_TCP are calculated by inverse kinematics (step S94g). It is determined whether the angle of each joint is within the movable range (step S94h). If the angle of each joint is within the movable range, (X_OBJ, Y_OBJ, and Z_TCP) are determined as the hand position (step S94i), and the selection process ends.

[0059] If the amount of rise exceeds the threshold in step S94f, or if the angle of each joint is outside the movable range in step S94h, it is determined whether there are any position candidates that have not yet been extracted (step S94j). If there are any position candidates that have not yet been extracted, the process is executed again for another position candidate. If there are no other position candidates, the hand position calculation process ends. This means that an object cannot be placed in the second container 42.

[0060] The advantages of the embodiment will be described. In picking operations using a robot, it is necessary to reduce the impact during transfer so as not to damage the object. To reduce the impact during transfer, it is preferable to acquire the size (three-dimensional shape) of the object in each of the X, Y, and Z directions. Based on the acquired three-dimensional shape, the object can be placed in the second space SP2 as the transfer destination without colliding with surrounding obstacles. In particular, by accurately acquiring the size of the object in the Z direction, it is possible to prevent the object from contacting obstacles when transferring or dropping when releasing the object, thereby reducing the impact on the object.

[0061] When multiple objects are placed in the first container 41 from which the objects are to be transferred, it is difficult to obtain an accurate three-dimensional shape from the measurement results obtained by the first measuring instrument 21. For example, if a portion of an object is hidden by another object, it is impossible to measure the length of the object in the Z direction. One possible method for obtaining the three-dimensional shape of the object to be transferred is to measure the object after the robot hand 11 acts on the object. The three-dimensional shape of the object can be obtained by exposing the hidden portion of the object through the action of the robot hand 11. However, calculating the hand position using the three-dimensional shape requires a large amount of calculation and is time-consuming. If the calculation time after the robot hand 11 acts on the object is long, the picking robot 10 must be stopped until the calculation result is obtained. This increases the time required for the picking operation and reduces work efficiency.

[0062] In the picking system 1 according to the embodiment, once a first measurement result is obtained by the first measuring instrument 21, a first calculation is performed to calculate position candidates for the object to be transferred. The position candidates are calculated based on the first measurement result from the first measuring instrument 21 and are candidates for the object's position in the second space SP2. As described above, it is difficult for the first measuring instrument 21 to accurately measure the length of the object to be transferred in the Z direction. However, if the first measurement result is available, it is possible to calculate candidate positions for the object, even if the final placement position of the object cannot be calculated. In other words, the first calculation can be started before the second measurement result is obtained from the second measuring instrument 22. Subsequently, in the picking system 1, once a second measurement result is obtained, the position of the robot hand 11 when placing the object in the second space SP2 is calculated based on the second measurement result and the position candidates. In other words, the second calculation is started after the first calculation and after the second measurement result is obtained. In the second calculation, the position candidates already calculated are used, so the hand position can be calculated in a shorter time.

[0063] According to the picking system 1, the hand position can be calculated at an earlier timing than when calculation of the object's placement position and hand position is started after the three-dimensional shape is obtained. This reduces the time the picking robot 10 has to stop to calculate the hand position. For example, the hand position can be calculated without stopping the picking robot 10. As a result, the picking work time can be reduced and work efficiency can be improved. Because the hand position can be calculated based on the three-dimensional shape of the object, it is possible to prevent the object from contacting an obstacle during transfer or falling when released, thereby reducing impact on the object. Furthermore, because the three-dimensional shape of the object is measured by the first measuring instrument 21 and the second measuring instrument 22, there is no need to prepare a three-dimensional model of the object in advance.

[0064] According to the embodiment, it is possible to reduce the time required for the picking operation while reducing the impact on the object during transfer.

[0065] The above describes an example in which the Z direction is parallel to the vertical direction, and the X and Y directions are parallel to the horizontal plane. The embodiment is not limited to this example. For example, the Y direction may be parallel to the vertical direction, and the X and Z directions may be parallel to the horizontal plane. In either case, the first measuring instrument 21 and the second measuring instrument 22 measure the shape of the object from different directions. A first calculation is performed after measurement by the first measuring instrument 21 and before measurement by the second measuring instrument 22, and a second calculation is performed after measurement by the second measuring instrument 22. This reduces the impact on the object during transfer and shortens the time required for the picking operation.

[0066] FIG. 10 is a schematic diagram showing a hardware configuration. The control device 30 includes, for example, the hardware configuration shown in Fig. 10. The processing device 90 shown in Fig. 10 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0067] The ROM 92 stores programs that control the operation of the computer. The ROM 92 stores programs necessary for the computer to execute each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0068] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a work memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During program execution, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0069] The storage device 94 stores data necessary for executing the program and data obtained by executing the program.

[0070] The input interface (I / F) 95 connects the processing device 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0071] The output interface (I / F) 96 connects the processing device 90 and the output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI (registered trademark)). The CPU 91 can transmit data to the output device 96a via the output I / F 96 and cause the output device 96a to display an image.

[0072] The communication interface (I / F) 97 connects the processing device 90 to a server 97a external to the processing device 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. Images taken by the photographing units 21a and 23a and detection results by the light curtain 22a are saved in the server 97a.

[0073] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (voice input), and a touchpad. The output device 96a includes one or more selected from a monitor and a projector. A device having the functions of both the input device 95a and the output device 96a, such as a touch panel, may also be used.

[0074] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0075] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, a computer reads a program from the recording medium and causes a CPU to execute instructions written in the program based on the program. The computer may acquire (or read) the program via a network.

[0076] According to the embodiments described above, a picking system, a control device, a picking method, a program, and a storage medium are provided that can reduce the time required for picking operations while reducing the impact on objects during transfer.

[0077] Although several embodiments of the present invention have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, the above-described embodiments can be implemented in combination with each other. [Explanation of symbols]

[0078] 1: Picking system, 10: Picking robot, 11: Robot hand, 12: Robot arm, 13: Housing, 21: First measuring instrument, 21a: Photography unit, 22: Second measuring instrument, 22a: Light curtain, 22a1: Light projecting unit, 22a2: Light receiving unit, 23: Third measuring instrument, 23a: Photography unit, 30: Control device, 31: Integration unit, 32: Measurement information processing unit, 33: Holding plan generation unit, 34: Layout plan generation unit, 34a: Position candidate calculation unit, 34b: Hand position calculation unit, 35: Motion plan generation unit, 36: Robot control device, 37: External I / F, 41: First container, 42: Second container, 90: Processing unit, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 95a: Input device, 96: Output interface, 96a: Output device, 97: Communication interface, 97a: Server, 98: System bus, OP1: First opening, OP2: Second opening, SP1: First space, SP2: Second space, SR: Sensing area

Claims

1. a picking robot that transfers an object from the first space to the second space using a robot hand; a control device that controls the picking robot, The control device acquiring a first measurement result obtained by measuring a shape of an object placed in the first space along a first direction; performing a first calculation to calculate position candidates of the object in the first direction, the second direction, and the third direction in the second space by determining whether or not the object is in contact with another object using the first measurement result while shifting a position of the object in the second space by a predetermined amount in the first direction, a second direction intersecting with the first direction, and a third direction intersecting with a plane including the first direction and the second direction; acquiring a second measurement result obtained by measuring the shape of the object along the second direction while the robot hand is acting on the object; performing a second calculation to calculate a position of the robot hand when the object is placed in the second space by adding the protrusion amount of the object in the first direction indicated by the second measurement result to the position candidate; Picking system.

2. a first measuring instrument that measures the shape of the object along the first direction to obtain the first measurement result; a second measuring instrument that measures the shape of the object along the second direction to obtain the second measurement result; The picking system of claim 1 further comprising:

3. A picking system as described in claim 2, wherein the second measuring instrument measures the shape of the object along the second direction after the first measurement result is obtained by the first measuring instrument.

4. The picking system according to claim 2 or 3, wherein the second measuring instrument measures the shape of the object while the object is being held by the picking robot.

5. 5. The picking system according to claim 2, wherein the first measuring instrument measures the shape of a first surface of the object that intersects with the first direction.

6. The picking system according to claim 5 , wherein the first measuring instrument measures the shape of the object on the first surface by segmenting a region of the object from the image of the first space.

7. 7. The picking system according to claim 2, wherein the second measuring instrument measures the amount of protrusion of the object from the tip of the robot hand in the first direction.

8. the second measuring instrument has a sensing area extending along a first plane intersecting the first direction; The picking system according to claim 7 , wherein the second measuring instrument measures the protrusion amount based on the time when the object passes through the sensing area while the picking robot is moving the object in the first direction.

9. The picking system according to claim 8 , wherein the second measuring instrument measures the protrusion amount based on the position of the sensing area in the first direction and the position of the robot hand in the first direction at the time.

10. further comprising a third measuring device that measures the shape of the second space; The picking system according to any one of claims 1 to 8, wherein the control device, in the first calculation, calculates the position candidate based on the first measurement result and the third measurement result from the third measuring instrument.

11. the third measuring device measures a shape of an obstacle in the second space; The control device In the first calculation, the position candidate is calculated based on the shape of the object and the shape of the obstacle; In the second calculation, the position of the robot hand is calculated based on the shape of the object, the shape of the obstacle, and the position candidate. The picking system according to claim 10.

12. The control device In the first calculation, a plurality of the position candidates are calculated; In the second calculation, the position of the robot hand is calculated using one of the plurality of position candidates and the protrusion amount. A picking system according to any one of claims 1 to 11.

13. the first direction is parallel to the vertical direction, The picking system according to any one of claims 1 to 12, wherein the second direction is parallel to the horizontal direction.

14. A control device that controls a picking robot that transfers an object from a first space to a second space using a robot hand, acquiring a first measurement result obtained by measuring a shape of an object placed in the first space along a first direction; performing a first calculation to calculate position candidates of the object in the first direction, the second direction, and the third direction in the second space by determining whether or not the object is in contact with another object using the first measurement result while shifting a position of the object in the second space by a predetermined amount in the first direction, a second direction intersecting with the first direction, and a third direction intersecting with a plane including the first direction and the second direction; acquiring a second measurement result obtained by measuring the shape of the object along the second direction while the robot hand is acting on the object; performing a second calculation to calculate a position of the robot hand when the object is placed in the second space by adding the protrusion amount of the object in the first direction indicated by the second measurement result to the position candidate; Control device.

15. A picking method using a picking robot that transfers an object from a first space to a second space using a robot hand, acquiring a first measurement result obtained by measuring a shape of an object placed in the first space along a first direction; performing a first calculation to calculate position candidates of the object in the first direction, the second direction, and the third direction in the second space by determining whether or not the object is in contact with another object using the first measurement result while shifting a position of the object in the second space by a predetermined amount in the first direction, a second direction intersecting with the first direction, and a third direction intersecting with a plane including the first direction and the second direction; acquiring a second measurement result obtained by measuring the shape of the object along the second direction while the robot hand is acting on the object; performing a second calculation to calculate a position of the robot hand when the object is placed in the second space by adding the protrusion amount of the object in the first direction indicated by the second measurement result to the position candidate; Picking method.

16. A program for causing a computer to control a picking robot that transfers an object from a first space to a second space using a robot hand, The computer, acquiring a first measurement result obtained by measuring a shape of an object placed in the first space along a first direction; a first calculation is executed to calculate position candidates of the object in the first direction, the second direction, and the third direction in the second space by determining whether or not the object is in contact with another object using the first measurement result while shifting a position of the object in the second space by a predetermined amount in the first direction, a second direction intersecting with the first direction, and a third direction intersecting with a plane including the first direction and the second direction, acquiring a second measurement result obtained by measuring the shape of the object along the second direction while the robot hand is acting on the object; performing a second calculation to calculate a position of the robot hand when the object is placed in the second space by adding a protrusion amount of the object in the first direction indicated by the second measurement result to the position candidate; program.

17. A storage medium storing the program according to claim 16.

Images