Handling system, transport system, control device, program, and handling method

Abstract

To provide a handling system which enables improvement of throughput of a system, and to provide a transport system, a control device, a program, and a handling method.SOLUTION: A handling system enables handling of multiple objects and has: movable arms; holding parts; sensors; and a control unit. Each holding part is attached to the movable arm and may hold the objects. The sensor may detect the objects. The control unit controls the movable arms and the holding parts. The control unit determines whether or not arrangement of the objects is changed based on information acquired from the sensors before the holding part holds the objects. When it is determined that the arrangement of the objects is changed, the control unit evaluates effectiveness of arrangement change operation for each object and determines the arrangement change operation based on an evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a handling system, a transport system, a control device, a program, and a handling method. [Background technology]

[0002] Conventionally, handling devices in which an end effector holds an object are known. Automating transfer operations in logistics sites requires the ability to hold objects of a wide variety of shapes, sizes, and weights. When holding these objects using a handling device, many calculations are required to determine the holding strategy, including the holding position, holding method, and robot arm posture. The more complex the object loading state, the longer the calculation time required to determine the holding strategy.

[0003] Furthermore, when performing a task using multiple holding tools selectively, switching between holding methods too frequently can increase the time required for the entire task by the amount of time required for tool replacement. Furthermore, in situations where there is a high probability of failure in a holding operation, such as when objects to be held are densely packed, the time required for the entire task can be increased by the number of holding failures. As a result, the system throughput can decrease. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6016716 [Patent Document 2] Patent No. 3925020 Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a handling system, a transport system, a control device, a program, and a handling method that improve the throughput of the system. [Means for solving the problem]

[0006] A handling system according to an embodiment is a handling system capable of handling a plurality of objects, and includes a movable arm, a holding unit, a sensor, and a control unit. The holding unit is attached to the movable arm and is capable of holding the object. The sensor is capable of detecting the object. The control unit controls the movable arm and the holding unit. The control unit determines, based on information acquired from the sensor, whether to rearrange the object before holding it, and if it determines to rearrange the object, evaluates the effectiveness of the rearrangement operation for each object and determines the rearrangement operation based on the results of the evaluation. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a perspective view schematically showing a transport system including a handling system according to a first embodiment. [Figure 2] 1 is a block diagram showing the system configuration of a transport system including a handling system according to a first embodiment. [Figure 3] 3 is a control flowchart of the control device according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing a temporary mask area in the image data of an object when the object O is held by "clamping." [Figure 5] FIG. 10 is a diagram showing a mask region in image data of an object. [Figure 6] FIG. 1 is a diagram showing a depth image of an object. [Figure 7] 1A and 1B are diagrams illustrating the three-dimensional position and orientation of an object. [Figure 8] FIG. 10 is a diagram showing a mask region to which information about three-dimensional position and orientation has been added. [Figure 9] 4 is a control flowchart of a retention strategy planning process performed by the control device according to the first embodiment. [Figure 10] FIG. 10 is a schematic diagram showing an example of a layout change operation. [Figure 11] FIG. 10 is a schematic diagram showing an example of a layout change operation. [Figure 12] FIG. 10 is a schematic diagram showing an example of a layout change operation. [Figure 13] FIG. 10 is a schematic diagram showing an example of a layout change operation. [Figure 14] 10 is a control flowchart of a control device according to a second embodiment. [Figure 15] 10 is a control flowchart of a retention strategy planning process performed by a control device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, a handling system, a transport system, a control device, a program, and a handling method according to embodiments will be described with reference to the drawings. In the following description, components having the same or similar functions are assigned the same reference numerals. Duplicate descriptions of those components may be omitted. In addition, "based on XX" in this application means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0009] (First embodiment) An embodiment will be described with reference to Figures 1 to 13. Figure 1 is a perspective view that schematically shows a transport system 1 including a handling device 10 (an example of a "handling system") of this embodiment.

[0010] The transport system 1 is, for example, a handling system (picking system) for logistics. The transport system 1 moves objects (objects to be held, objects to be transported) O located at a source V1 to a destination V2. For example, the transport system 1 performs the task of picking out a specified number of various types of objects O stored at the source V1 or the like and loading them at the destination V2.

[0011] The source V1 may be, for example, various conveyors, various pallets, or containers such as totes or container containers. A "container" broadly refers to a component (e.g., a box-shaped component) that can accommodate an object O. However, the source V1 is not limited to the above examples. In the following description, the "source V1" may be referred to as the "source container V1."

[0012] Many types of objects O of different sizes and weights are placed randomly at the source V1. For example, the object O to be held has an uneven shape on at least a portion of its surface. In this embodiment, the external shape of the object O varies from small objects such as 5 cm square to large objects such as 30 cm square. Furthermore, the object O varies from light objects such as several tens of grams to heavy objects such as several kilograms. However, the size and weight of the object O are not limited to the above example.

[0013] The destination V2 is, for example, a container such as a tote bag or an OriGan container. However, the destination V2 is not limited to the above example. In the following description, the "destination V2" will be referred to as the "destination container V2," and the "source V1" and the "destination V2" may be collectively referred to simply as the "container." Note that the transport system 1 may also transport the object O to a destination V2 other than a container.

[0014] The conveying system 1 is not limited to a handling system for logistics. The conveying system 1 can also be widely applied to industrial robot systems and other systems. In this application, the terms "conveying system," "handling system," and "handling device" are not limited to systems and devices whose main purpose is to transport objects, but also include systems and systems that involve transporting (moving) objects as part of product assembly or other purposes.

[0015] 1, the transport system 1 includes a handling device 10, a sensor 11, and a control device 12 (an example of a "control unit"). The control device 12 may be incorporated into the handling device 10.

[0016] The handling device 10 is, for example, a robot device. The handling device 10 holds an object O located in a source container V1 and moves the held object O to a destination container V2 (storage area). The handling device 10 can communicate with a control device 12 via wire or wirelessly. In this embodiment, the handling device 10 has a first handling device 10A and a second handling device 10B.

[0017] The first handling device 10A has, for example, a movable arm 100 and a first holding part 200A provided at the tip of the movable arm 100.

[0018] The movable arm 100 is a movement mechanism that moves the first holding unit 200A to a desired position. For example, the movable arm 100 is a six-axis vertical articulated robot arm. The movable arm 100 can assume a variety of positions and postures. Like a human arm or hand, the movable arm 100 can also assume a variety of postures for holding an object. The movable arm 100 includes, for example, a plurality of arm members 101 and a plurality of rotating units 102 that rotatably connect the plurality of arm members 101.

[0019] The movable arm 100 may be a three-axis Cartesian robot arm. The movable arm 100 may be a mechanism that moves the first holding unit 200A to a desired position using other configurations. For example, the movable arm 100 may be an aircraft (e.g., a drone) that lifts and moves the first holding unit 200A using rotating wings.

[0020] The first holding unit 200A is a holding mechanism (end effector) that holds the object O located in the source container V1. The first holding unit 200A has a clamping hand 202.

[0021] The clamping hand 202 is a gripper-type hand that pinches and grasps the object O between two fingers, and is provided at the tip of the movable arm 100. The configuration of the clamping hand 202 is not limited to this, and may be, for example, a gripper-type hand that pinches and grasps the object O between three fingers.

[0022] The first holding unit 200A may be a hybrid hand that further includes a suction device and an adsorption unit that communicates with the suction device and that holds the object O by clamping and / or adsorption. In this case, the adsorption unit may be provided on the fingertips of the clamping hand 202. A plurality of adsorption units may be provided on the fingertips of the clamping hand 202.

[0023] The second handling device 10B is, for example, movable The second handling device 10B has an arm (second arm) 100 and a second holding part 200B provided at the tip of the movable arm 100. The movable arm 100 of the second handling device 10B has the same configuration as the movable arm 100 of the first handling device 10A.

[0024] The second holding unit 200B is a holding mechanism (end effector) that holds the object O located in the source container V1. For example, the second holding unit 200B has a suction device 203 and a suction unit 205 that communicates with the suction device 203. The second holding unit 200B is a suction-type hand that holds the object O by suction.

[0025] The second holding unit 200B may be a mechanism that holds the object O using other holding methods. For example, the second holding unit 200B may be a holding unit that can hold the object O using magnetic force. For example, the second holding unit 200B may be a holding unit (e.g., a jamming gripper) that is composed of a flexible membrane filled with powder and a vacuum pump that removes air from inside the flexible membrane and that can hold the object O using a jamming phenomenon.

[0026] The suction device 203 is, for example, a vacuum pump. The suction device 203 is in communication with each of the plurality of suction units 205 via a hose or the like. When the suction device 203 is driven, the pressure inside each suction unit 205 becomes lower than atmospheric pressure, and the object O is sucked and held by the suction unit 205.

[0027] The suction unit 205 is provided at the tip of the second holding unit 200B. For example, a plurality of suction units 205 are provided at the tip of the second holding unit 200B. The suction units 205 have an outer shape smaller than the smallest object O located in the source container V1. The second handling device 10B suction-holds the object O using only one or more suction units 205 selected from the plurality of suction units 205.

[0028] In the following description, the "first holding unit 200A" and the "second holding unit 200B" will be collectively referred to as "holding unit 200." In other words, "holding unit 200" will include the "first holding unit 200A" and the "second holding unit 200B." Note that, here, the first holding unit 200A has been described as a clamping hand and the second holding unit 200B as a suction hand, but the configuration of holding unit 200 is not limited to one having one first holding unit 200A of a clamping hand and one second holding unit 200B of a suction hand as described above. In this embodiment, the case where both the first holding unit 200A and the second holding unit 200B are clamping hands or suction hands is also included. In this case, 、 The holding unit 200 may be configured to have multiple clamping hands that differ in at least one of the characteristics, such as configuration, structure, shape, dimensions, and arrangement. Specifically, for example, the first holding unit 200A and the second holding unit 200B may be two or more clamping hands with different claw lengths or opening widths. The holding unit 200 may also be configured to have multiple suction hands that differ in at least one of the characteristics, such as configuration, structure, shape, dimensions, and arrangement. Specifically, for example, the first holding unit 200A and the second holding unit 200B may be two or more suction hands with different suction pad arrangements, suction pad diameters, bellows structures, etc. Even in these cases, the present embodiment can be implemented in the same way to obtain the same effects.

[0029] The sensor 11 is controlled by the control device 12 to detect the states of the multiple objects O and / or the holder 200. The sensor 11 includes a first sensor 11A, a second sensor 11B, a third sensor 11C, a fourth sensor 11D, and a fifth sensor 11E. The first sensor 11A, the second sensor 11B, the third sensor 11C, the fourth sensor 11D, and the fifth sensor 11E are connected to the control device 12 by wire or wirelessly. Note that the first sensor 11A to the fifth sensor 11E do not necessarily have to be separate sensors, and a specific sensor may independently perform the functions of multiple sensors among the first sensor 11A to the fifth sensor 11E.

[0030] The first sensor 11A is a camera or various sensors arranged near the source V1 (for example, directly above or diagonally above the source V1). The first sensor 11A acquires, for example, information about the object O located at the source V1 and information about the source V1. The information acquired by the first sensor 11A includes, for example, "image data," "distance image data," and "shape data." "Distance image data" is image data having distance information in one or more directions (for example, depth information from an arbitrary reference plane set above the source V1). "Shape data" is information indicating the outer shape of the object O, etc. The information detected by the first sensor 11A is output to the control device 12. The first sensor 11A may be provided as part of the handling device 10.

[0031] The second sensor 11B is a camera or various sensors arranged near the destination container V2 (for example, directly above or diagonally above the destination container V2). The second sensor 11B detects, for example, information about the shape of the destination container V2 (including the shapes of the inner wall surfaces and partitions) and information about the object O previously placed in the destination container V2. The information acquired by the second sensor 11B includes, for example, "image data," "distance image data," and "shape data." The information detected by the second sensor 11B is output to the control device 12. The second sensor 11B may be provided as part of the handling device 10.

[0032] The third sensor 11C is a type of sensor provided on or near the first holding unit 200A. The third sensor 11C acquires information about the physical state of the first holding unit 200A, such as the distortion of the first holding unit 200A, the pressure applied to the first holding unit 200A, and the surface condition of the first holding unit 200A. The third sensor 11C includes one or more physical sensors, such as a distortion sensor, a pressure sensor, and a proximity sensor. The third sensor 11C may further acquire physical information about the object O. The information detected by the third sensor 11C is output to the control device 12. The third sensor 11C may be provided as part of the handling device 10.

[0033] The fourth sensor 11D is a sensor provided on or near the second holding unit 200B. The fourth sensor 11D acquires information about the physical state of the second holding unit 200B, such as the distortion of the second holding unit 200B, the pressure applied to the second holding unit 200B, and the surface condition of the second holding unit 200B. The fourth sensor 11D includes one or more physical sensors, such as a distortion sensor, a pressure sensor, and a proximity sensor. The fourth sensor 11D may also acquire physical information about the object O. The information detected by the fourth sensor 11D is output to the control device 12. The fourth sensor 11D may be provided as part of the handling device 10.

[0034] The fifth sensor 11E acquires information regarding the usage status of the holding unit 200. For example, the fifth sensor 11E detects the holding unit 200 currently being used or selected for use out of the first holding unit 200A and the second holding unit 200B (hereinafter, the holding unit currently being used or selected for use may be collectively referred to as the "currently selected holding unit" or simply the "selected holding unit"). The information detected by the fifth sensor 11E is output to the control device 12. The fifth sensor 11E may be provided as part of the handling apparatus 10. The currently selected holding unit 200 may also be determined based on other information, such as the control history of the handling apparatus 10 or information acquired by the third sensor 11C or the fourth sensor 11D, rather than the fifth sensor 11E. In this case, the fifth sensor 11E may be omitted. In addition, if the holding unit 200 is a hybrid hand capable of clamping, adsorption, and both clamping and adsorption, the control device 12 determines, based on information from the fifth sensor 11E or other information, whether the holding unit 200 is in a state suitable for clamping the object O, for adsorption, or for both.

[0035] The control device 12 manages and controls the entire transport system 1. For example, the control device 12 acquires information detected by the first sensor 11A to the fifth sensor 11E, and controls the handling device 10 based on the acquired information. The control device 12 is, for example, a programmable device (computer) equipped with a processor, memory, storage, etc.

[0036] FIG. 2 is a block diagram showing the system configuration of the transport system 1. As shown in FIG. The control device 12 is directly or indirectly connected by wire or wirelessly to the first sensor 11A to the fifth sensor 11E, the movable arm 100, the first holding unit 200A, and the second holding unit 200B. The control device 12 includes an input unit 300, a recognition processing unit 301, a memory unit 302, an operation control unit 303, a score calculation unit 304, a threshold generation unit 305, a judgment unit 306, a holding strategy determination unit 307, a holding inhibition factor estimation unit 308, and a placement change strategy determination unit 309.

[0037] The input unit 300 receives an order list relating to the object O to be held from an operator or the system, information acquired by the first sensor 11A to the fifth sensor 11E, and the like.

[0038] The recognition processing unit 301 processes information acquired by the first sensor 11A to the fifth sensor 11E. For example, the recognition processing unit 301 determines the position, posture, shape, characteristics, etc. of the object O at the origin V1 based on the image data acquired by the first sensor 11A.

[0039] The memory unit 302 records the control program for the control device 12, shape data on the object O to be picked, the order list received from the operator or the system, various scores generated by the score calculation unit 304, thresholds generated by the threshold generation unit 305, the determination result by the determination unit 306, the holding strategy determined by the holding strategy determination unit 307, the holding impediment factors estimated by the holding impediment factor estimation unit 308, the rearrangement strategy determined by the rearrangement strategy determination unit 309, the control history and operation history of the handling device 10, etc. The shape data stored in the memory unit 302 is defined in the local coordinate system of the object O.

[0040] The operation control unit 303 controls the operation of the movable arm 100, first holding unit 200A, second holding unit 200B, etc. of the handling device 10. For example, the operation control unit 303 specifically calculates a position at which to hold an object O and the posture of the movable arm 100 when holding the object O, which are appropriate for holding a specific object O using a specific holding means. The operation control unit 303 instructs the handling device 10 to cause the first holding unit 200A or the second holding unit 200B to perform a holding operation of the specific object O based on the calculated position at which to hold the object O and the posture of the movable arm 100 when holding the object O. The operation control unit 303 also specifically calculates a position at which to change the position of the object O and the posture of the movable arm 100 when changing the position, which are appropriate for changing the position of the specific object O using a specific holding means. The operation control unit 303 instructs the handling device 10 to have the first holding unit 200A or the second holding unit 200B perform a repositioning operation on a specific object O based on the calculated position at which the object O will be repositioned and the posture of the movable arm 100 when the repositioning is performed.

[0041] In this specification, the term "rearrangement operation" refers to a physical operation that changes the spatial arrangement of a specific object by using a physical force. The content of the rearrangement operation is specified by the object, the specific force applied to the object, the means used to apply the force to the object, the specific manner in which the spatial arrangement of the object is changed, etc.

[0042] The score calculation unit 304 generates scores for determining the object O to be held, the priority of the holding method, the object O to be rearranged, the effectiveness of the rearrangement operation, etc. For example, the score calculation unit 304 calculates a first score, a second score, and a third score, which will be described later.

[0043] The threshold generation unit 305 generates thresholds for determining whether or not it is necessary to switch the holding method, whether or not it is necessary to change the arrangement, etc. For example, the threshold generation unit 305 generates a first threshold for a first score and a second threshold for a second score, which will be described later.

[0044] The determination unit 306 determines whether or not it is necessary to switch the holding method, whether or not it is necessary to change the placement of the object O, etc., based on the score generated by the score calculation unit 304, the threshold generated by the threshold generation unit 305, etc.

[0045] The retention strategy determination unit 307 determines a retention strategy including high-priority retention objects and retention methods, the order in which retention objects are retained, and the order in which retention operations and retention method switching operations are performed, based on the score generated by the score calculation unit 304, the threshold generated by the threshold generation unit 305, the result of judgment by the judgment unit 306, etc.

[0046] Based on the information acquired by the first sensor 11A to the fifth sensor 11E, the holding impediment factor estimation unit 308 estimates the presence or absence of various holding impediment factors that impede holding of the object O by the holding unit 200. For example, the holding impediment factor estimation unit 308 estimates the presence or absence of holding impediment factors based on the image data acquired by the first sensor 11A and the recognition result output by the recognition processing unit 301.

[0047] The placement change strategy determination unit 309 determines a placement change strategy for changing the placement of the object O so that it becomes easier to hold the object O when it is difficult to hold the object O, based on the score generated by the score calculation unit 304, the threshold generated by the threshold generation unit 305, the result of the judgment by the judgment unit 306, etc.

[0048] Some or all of the functions of the control device 12 are realized by one or more processors, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), executing programs stored in a program memory. However, some or all of these functions may be realized by hardware (e.g., circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a PLD (Programmable Logic Device). Some or all of the functions may also be realized by a combination of software and hardware. The storage unit 302 is realized by a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a RAM (Random Access Memory), or the like.

[0049] Next, the operation of the transport system 1 will be described with reference to the control flowchart of the control device 12 shown in FIG.

[0050] When the control device 12 is started, the control device 12 initializes the handling device 10 and the sensor 11, and then starts controlling the handling device 10 (step S0). Next, the control device 12 executes step S1.

[0051] In step S1, the input unit 300 of the control device 12 receives an order list of objects O to be picked from an operator or the system (order list receiving step).

[0052] Next, the control device 12 executes step S2. In step S2, the input unit 300 of the control device 12 receives image data, range image data, shape data, etc. related to the object O in the source container V1 from the first sensor 11A. The recognition processing unit 301 of the control device 12 determines whether the object O on the order list is present in the source container V1 based on the data received by the input unit 300. Furthermore, the recognition processing unit 301 acquires information related to the shape, position, orientation, etc. of the object O to be picked based on the data received by the input unit 300 (information acquisition process).

[0053] FIG. 4 is a diagram showing a temporary mask region R1 in image data of the object O when the object O is held by "pinching." The recognition processing unit 301 of the control device 12 uses a known image segmentation method to set a rectangular area (circumscribed rectangular area) circumscribing the object O to be picked from the image data as a "temporary mask area R1." Image segmentation may be a method using machine learning.

[0054] FIG. 5 is a diagram showing a mask region R2 in the image data of the object O. The recognition processing unit 301 sets an area obtained by extending the temporary mask area R1 in the vertical and horizontal directions as a "mask area R2." The mask area R2 is extended by a margin M in the vertical and horizontal directions of the circumscribing rectangular area. For example, the margin M is 100 mm. By using the extended mask area R2, the recognition processing unit 301 determines whether the clamping hand 202 is positioned around the object O when the holding method by the holding unit 200 is clamping. approach You can determine if there is space available.

[0055] FIG. 6 is a diagram showing a depth image D of an object O. As shown in FIG. The recognition processing unit 301 uses the distance image data to generate a depth image D that visualizes the depth information of the object O in the mask region R2. The depth image D has a height value based on the origin of the world coordinate system (X-axis, Y-axis, Z-axis). The scale of the depth image D can be changed depending on the storage method used by the storage unit 200. The scale of the depth image D can be set to, for example, 1 mm per pixel.

[0056] Fig. 7 is a diagram illustrating the three-dimensional position and orientation of an object O. In Fig. 7, objects O with different shapes are denoted as "O1" and "O2." The recognition processing unit 301 calculates the three-dimensional position and orientation of the object O1 from the acquired image data of the object O1. The recognition processing unit 301 converts the shape data of the object O1 in the local coordinate system recorded in the storage unit 302 into a world coordinate system (X-axis, Y-axis, Z-axis) using a transformation matrix. The Z-axis direction of the world coordinate system is the depth direction from an arbitrary reference plane set above the source container V1, as shown in FIG. 7. The recognition processing unit 301 calculates the three-dimensional position and orientation of the object O1 by comparing the acquired image data of the object O1 with the shape data converted into the world coordinate system.

[0057] 8 is a diagram showing a mask area R2 to which information regarding the three-dimensional position and orientation has been added. The recognition processing unit 301 sets the three-dimensional position pose serving as the reference of the object O1 as the center CO of a circumscribing rectangular area (temporary mask area R1) in the mask image R2. Furthermore, the recognition processing unit 301 may calculate and use the centroid FO of the object O1 separately from the center CO of the circumscribing rectangular area, taking into account the ease of gripping by the gripping hand 202.

[0058] On the other hand, when holding object O by "suction," the temporary mask region R1 is divided into mask planes on the surface of object O that can be targets for suction. The recognition processing unit 301 generates a mask image R2 and a depth image D for each mask plane and defines the direction perpendicular to the mask plane as the normal direction. The recognition processing unit 301 can, for example, extract a plane region from the point cloud of a 3D sensor and determine the position and orientation by defining the minor axis direction as x, the major axis direction as y, and the normal direction of the plane as z, for example, using principal component analysis. In the case of "holding," only one holding region may be defined for one object O, but in the case of "suction," multiple holding regions may exist for one object O. Therefore, the holding target in "suction" may be a region of object O. Note that, for example, if there are multiple holding regions for "holding," the holding target in "holding" may also be a region of object O.

[0059] When holding object O by "adhesion," the recognition processing unit 301 can perform convolution processing to calculate how many of the multiple adsorption units 205 to use, at what angle the adsorption units 205 should contact object O, etc.

[0060] The recognition processing unit 301 may use a database that records the three-dimensional position and orientation of the object O when the object O has been successfully picked up in the past. The recognition processing unit 301 can also use the database to output a recommended holding method and holding position for the object O.

[0061] The input unit 300 receives information about the physical state of the first holding unit 200A from the third sensor 11C and information about the physical state of the second holding unit 200B from the fourth sensor 11D as needed. Furthermore, the input unit 300 receives information about the usage status of the holding units 200, such as the currently selected holding unit, from the fifth sensor 11E as needed. The recognition processing unit 301 determines whether the first holding unit 200A or the second holding unit 200B is the currently selected holding unit 200 based on the information from the fifth sensor 11E and / or other information. The information about the physical states of the first holding unit 200A and the second holding unit 200B and the information about the currently selected holding unit 200 may be acquired at a timing other than step S2, as appropriate.

[0062] Furthermore, the input unit 300 receives physical information of the object O to be stored from a database. For example, the input unit 300 can receive information on the external shape, weight, surface material, frictional characteristics, etc. of the object O from the database. The database may be defined for each individual object O, or may define basic information primitives (e.g., rectangular parallelepiped, cylinder, sphere, square pyramid, etc.) and use information that approximates the shape.

[0063] Next, the control device 12 executes step S3. In step S3, the control device 12 calculates a score for each object O or region of the object O and for each holding method, and determines the next object O to be held and the holding method (holding strategy planning step). If holding is predicted to be difficult, the control device 12 determines the content of the object O's placement change operation. FIG. 9 is a control flowchart of the holding strategy planning step by the control device 12, and shows the details of step S3.

[0064] In step S301, the score calculation unit 304 of the control device 12 calculates the ease with which the holding unit 200 holds the object O as a "first score" based on the information acquired in step S2. The score calculation unit 304 calculates a first score (1) for each object O or region of the object O (hereinafter simply referred to as "each object O"), and (2) for each holding method. For example, the holding method is "clamping" by the first handling device 10A or "suction" by the second handling device 10B. When the holding method is "clamping," the first score is calculated, for example, for each object O, and when the holding method is "suction," the first score is calculated, for example, for each region of the object O.

[0065] The first score S calculated in step S301 H (I) is calculated, for example, by the evaluation function shown in Equation 1.

number

[0066] In Equation 1, H is the holding method (such as clamping by the first handling device 10A or suction by the second handling device 10B) that is the evaluation target of the first score. I is the mask region R2 that indicates the object O or the region of the object O that is the evaluation target of the first score. Hi is the evaluation item of the evaluation function, and w Hi is the weight of the evaluation function. In other words, the above evaluation function is expressed as a linear combination of the evaluation items. However, the evaluation function is not limited to the above example, and an average value of the evaluation items may be used, or a nonlinear function may be used, and any evaluation function can be used. Furthermore, the evaluation items are not limited to the items described in Equation 1, and may be, for example, quantities dependent on the physical properties of the object O or the clamping hand 202.

[0067] When the holding method is "clamping" (H=p), the first score S p (I) is calculated, for example, by the evaluation function shown in Equation 2.

number

[0068] In Equation 2, w p1 +w p2 +w p3 +w p4 +w p5 +w p6 = 1, and w pi ≧0(i=1,2,3,4,5,6). f p1 is the position of the object O (for example, the center position and height of the object O). p2 is the roughness of the object O. f p3 is the flatness of object O. f p4 is the depth difference between object O and its surroundings. p5 is the result of determining whether the object is thin or not. p6 is an evaluation item used when using machine learning, and the above f p1 ~f p5 For example, f p6 When using w p1 ~w p5 can be set to zero. For various mask regions I representing each object O, the first score S of "holding" is p (I) is calculated.

[0069] If the holding method is "adsorption" (H=s), the first score S s (I) is calculated, for example, by the evaluation function shown in Equation 3.

number

[0070] In Equation 3, w s1 +w s2 +w s3 +w s4 +w s5 +w s6 = 1, and w si ≧0(i=1,2,3,4,5,6). f s1 is the position of the region of object O (for example, the center position of the region of object O and the height of object O). s2is the roughness of the area of the object O. f s3 is the area of the region of object O. f s4 is the approach angle to the area of the object O (for example, the angle between the normal direction of the area of the object O and the vertical direction). s5 is the result of determining whether the object is thin or not. s6 is an evaluation item used when using machine learning, and the above f s1 ~f s5 For example, f s6 When using w s1 ~w s5 can all be set to zero. The first score S of "adhesion" for various mask regions I representing the area of the object O s (I) is calculated.

[0071] In this way, the score calculation unit 304 calculates a first score S for each holding method (here, clamping by the first handling device 10A and suction by the second handling device 10B) for each object O or region of the object O. H Calculate (I).

[0072] Next, the control device 12 executes step S302. In step S302, the determination unit 306 of the control device 12 determines the first scores S calculated in step S301. H It is determined whether or not there is a first score S that is equal to or greater than a predetermined first threshold value Th1 among (I). H If it is determined that at least one of (I) is equal to or greater than the first threshold Th1 (step S302: Yes), the process of the control device 12 proceeds to step S303 to plan a holding strategy for the object O. On the other hand, all the first scores S calculated for each holding method and for each object O or region of the object O are H If it is determined that (I) is less than the first threshold value Th1 (step S302: No), the process of the control device 12 proceeds to step S310 to plan a relocation strategy that makes it easier to hold the object O.

[0073] In step S302, a first score S equal to or greater than a first threshold Th1 is calculated.H If it is determined that (I) exists (step S302: Yes), the control device 12 executes step S303. In step S303, the score calculation unit 304 of the control device 12 calculates a "second score" based on the information acquired in step S2 and the first score calculated in step S301, with the currently selected holding method as the reference. The score calculation unit 304 calculates the second score (1) for each object O or region of the object O, and (2) for each holding method.

[0074] The second score T calculated in step S303 H,H0 (I) is calculated, for example, by the evaluation function shown in Equation 4.

number

[0075] In Equation 4, H is the holding method to be evaluated for the second score. H0 is the currently selected holding method. I is the object O or the mask region R2 indicating the region of the object O to be evaluated for the second score. That is, the evaluation function of Equation 4 is calculated based on the first score S of the currently selected holding method. H0 (I) and the first score S of the retention method that is the evaluation target for the second score. H However, the evaluation function for the second score is not limited to the above example, and any function can be used. For example, the first score S of the currently selected retention method H0 is H0 (I) and the first score S of the retention method H that is the subject of evaluation of the second score H It can be any function that depends on (I).

[0076] 1st Score S H Since (I) is an index of the "ease of retention" of the target mask region I by the target retention method H, when using the evaluation function of the above formula 4, the second score T H,H0(I) is the ratio between the ease of retention by the currently selected retention method H0 and the ease of retention by the target retention method H. In other words, if the second score of Formula 4 is greater than 1, retention is easier by the currently selected retention method H0 than by the target retention method H. If the second score of Formula 4 is less than 1, retention is easier by the target retention method H0 than by the currently selected retention method H0. For example, if the second score of Formula 4 is 0.5, it can be said that the ease of retention by the currently selected retention method H0 is 0.5 times the ease of retention by the target retention method H. In other words, it can be said that the ease of retention by the target retention method H is twice the ease of retention by the currently selected retention method H0.

[0077] Next, the control device 12 executes step S304. In step S304, the determination unit 306 of the control device 12 determines the second score T H,H0 The necessity of switching the retention method is determined based on (I). H,H0 When (I) is used, for example, the judgment unit 306 judges that it is necessary to switch the holding method from the currently selected holding method H0 to the target holding method H if the conditional expression of the following mathematical expression 5 is satisfied, and judges that it is not necessary to switch the holding method from the currently selected holding method H0 to the target holding method H if the conditional expression of the mathematical expression 5 is not satisfied.

number

[0078] In Equation 5, Th2 is a predetermined second threshold generated by the threshold generating unit 305. That is, the determining unit 306 determines the second score T H,H0 If (I) is below the second threshold Th2, it is determined that the currently selected holding method H0 needs to be switched to the target holding method H, and a second score T H,H0 If (I) is equal to or greater than the second threshold value Th2, it is determined that there is no need to switch from the currently selected holding method H0 to the target holding method H.

[0079] 1st Score SH If you select the retention method only in (I), the information of the currently selected retention method H0 is used as the first score S H Since the second score T is not reflected in (I), the holding method with the highest success rate is simply selected. For this reason, in some cases, it may be determined that the holding method needs to be changed every time the object O is held, and in such cases, it may take time to change the holding method. On the other hand, H,H0 (I) is a score calculated based on the currently selected storage method H0, including information on the currently selected storage method H0. For example, the second score T H,H0 When using (I), the second score T H,H0 Even if (I) is smaller than 1 (i.e., the ease of holding by the target holding method H is greater than the ease of holding by the currently selected holding method H0), the determination unit 306 determines the second score T H,H0 If (I) is not so small (i.e., if it is equal to or greater than the predetermined second threshold value Th2), it is determined that there is no need to switch the holding method. As a result, the control device 12 can control the handling device 10 to use the currently selected holding method H0 as much as possible when the currently selected holding method H0 is reasonably easy to hold.

[0080] The determination unit 306 calculates a second score T for each mask region I that indicates the object O or the region of the object O to be retained. H,H0 The necessity of switching the retention method may be determined based on the magnitude relationship between the average value of (I) and a second threshold value Th2, and the second score T H,H0 The necessity of switching the holding method may be determined based on the magnitude relationship between each of the mask regions I and the second threshold Th2. For example, in the above example, the determination unit 306 may determine whether the second score T H,H0 If at least one of (I) is smaller than the second threshold value Th2, it can be determined that the holding method needs to be switched.

[0081] Next, when it is determined that switching of the holding method is not necessary (step S305: No), the control device 12 executes step S306. In step S306, the holding strategy determination unit 307 of the control device 12 determines a holding strategy including the order in which the objects O to be held are to be held. When the currently selected holding method H0 is "clamping", the holding strategy determination unit 307 determines the order in which the objects O are to be held based on, for example, the first score S for "clamping". p In addition, when the currently selected holding method H0 is "adhesion", the holding strategy determination unit 307 may, for example, determine the order of holding the object O in order of the first score S for "adhesion". s In other words, the retention strategy determination unit 307 may sort the data in descending order of the first score S without switching the currently selected retention method H0. H0 A holding strategy can be determined in which the object O is picked up by the currently selected holding method H0 in descending order of (I).

[0082] Next, the control device 12 executes step S308. In step S308, the control device 12 stores in the storage unit 302 the retention strategy including the retention order determined by the retention strategy determination unit 307 in step S306.

[0083] Next, the control device 12 executes step S309. In step S309, the retention strategy determination unit 307 of the control device 12 determines the next retention action. For example, the retention strategy determination unit 307 determines the earliest retention action in the retention strategy determined in step S306 (in the above example, the first score S H0 (Holding the object O with the largest I by the currently selected holding method H0) is set as the next holding operation. After that, the control device 12 proceeds to step S4.

[0084] On the other hand, if it is determined that switching of the holding method is necessary (step S305: Yes), the control device 12 executes step S307. In step S307, the holding strategy determination unit 307 determines a holding strategy including the order of holding and holding method switching. That is, the holding strategy determination unit 307 determines a holding strategy including the order in which holding operations and holding method switching operations should be performed on multiple objects O to be held. As an example, the holding strategy determination unit 307 can determine the order of holding and holding method switching as follows. (1) The first score S according to the currently selected retention method H0 H0 (I) is the first score S by the retention method H1 after switching H1 For an object O or a region of object O that is larger than (I), a first score S H0 The holding operation is performed in order of increasing (I). (2) The holding method is switched from the currently selected holding method H0 to the target holding method H1. (3) The first score S according to the currently selected retention method H0 H0 (I) is the first score S by the retention method H1 after switching H1 For an object O or a region of the object O that is smaller than (I), a first score S H1 The holding operation is performed in order of increasing (I).

[0085] This makes it possible to create a holding strategy that can pick up all of the objects O to be held by switching the holding method once, thereby minimizing the number of times the holding method is switched.

[0086] The method for determining the retention strategy is not limited to the above example. For example, the retention strategy determination unit 307 determines the second score T H1,H0 After the currently selected retention method H0 is used to retain the retention target whose (I) is equal to or greater than the second threshold Th2, the retention method is switched and the second score T H,H0 The retention strategy determination unit 307 may determine a retention strategy such that retention is performed by the retention method H1 after switching for retention targets for which (I) is less than the second threshold value Th2. The retention strategy determination unit 307 may determine a retention strategy by any other method.

[0087] Next, the control device 12 executes step S308. In step S308, the control device 12 stores in the storage unit 302 the retention strategy including the order of retention and retention method switching determined by the retention strategy determination unit 307 in step S307.

[0088] Next, the control device 12 executes step S309. In step S309, the retention strategy determination unit 307 of the control device 12 determines the next retention action. For example, the retention strategy determination unit 307 sets the earliest retention action in the retention strategy determined in step S307 as the next retention action. For example, in the example shown in (1) to (3) above, the retention strategy determination unit 307 determines the first score S by the currently selected retention method H0. H0 (I) is the first score S by the retention method H1 after switching H1 (I) is the first score S of the object O or the area of the object O that is larger than H0 The next retention operation can be set to retain the one with the largest (I) by the currently selected retention method H0. Note that the first score S by the currently selected retention method H0 H0 (I) is the first score S by the retention method H1 after switching H1 If there is no object O or area of the object O larger than (I), the retention strategy determination unit 307 determines the first score S according to the retention method H1 after switching. H1 It is possible to set the next holding operation to hold the object O or the area of the object O for which (I) is the largest using the holding method H1 after switching. After that, the control device 12 proceeds to step S4.

[0089] In the steps up to this point, the control device 12 preferably determines the next holding operation without specifically calculating the position where the object O is held or the posture of the movable arm 100.

[0090] In the above example, the control device 12 has been described as determining the entire order of holding and then selecting the earliest holding action among them as the next action, but it is also possible to omit determining the order of holding and determine only the next action. For example, the holding strategy determination unit 307 may determine the first score S without determining the order of holding in step S306. H The one with the largest (I) may be selected as the next action.

[0091] When determining the retention strategy in step S306 or step S307, the control device 12 uses the first score S H (I) or second score T H,H0 A corrected score obtained by correcting (I) for the weight of object O, etc. may also be used.

[0092] As mentioned above, the first score S is calculated by "holding" and "suction". H Just as the evaluation items for calculating (I) and the assumed retention target (object O or area of object O, etc.) differ, the first score S H The calculation criteria for (I) may vary. Therefore, the control device 12 may properly normalize the scores so that scores obtained through different retention methods can be compared. The control device 12 may perform the score normalization process at any step, such as step S301 for calculating the first score, step S303 for calculating the second score, or steps S306 and S307 for determining the retention strategy.

[0093] The details of the holding strategy planning in step S3 have been explained above. As an example, when the selectable holding methods are either "clamping" or "suction", the currently selected holding method H0 is "clamping", and the second threshold Th2 is set to 0.5, the second score T in the above formula 4 is calculated as follows: H,H0Examples of calculations by (I) are shown in Tables 1 to 3 below. Tables 1 to 3 are tables showing examples of score calculations. Tables 1 to 3 show, for holding targets 1 to 5 representing object O or a region of object O, a first score indicating the ease of holding by the holding method "clamping", a first score indicating the ease of holding by the holding method "adsorption", a second score calculated from these first scores, and a judgment result based on the second threshold value Th2. In the judgment result, "True" indicates that the second score T H,H0 (I) is smaller than the second threshold Th2 (i.e., the retention method needs to be switched), and "False" means that the second score T H,H0 This means that (I) is greater than the second threshold value Th2 (that is, it is not necessary to switch the holding method). [Table 1] [Table 2] [Table 3]

[0094] In the example shown in Table 1, the second score T H,H0 The average value of subjects 1 to 5 in (I) is 1.03, which is greater than 1. Therefore, it can be said that the ease of holding by the currently selected holding method H0 (clamping) is greater than the ease of holding by the holding method (suction) of the subject being evaluated. In addition, the second score T H,H0 Since (I) and its average value are both greater than the second threshold value Th2=0.5, the determining unit 306 of the control device 12 determines that it is not necessary to switch the holding method.

[0095] Next, the retention strategy determination unit 307 determines the order of retention based on the first score S H0The holding strategy is determined so that the objects are held in the order of "object 3 → object 2 and object 4 → object 1 and object 5" by the currently selected holding method, "sandwiching." Next, the control device 12 stores the determined holding strategy in the memory unit 302. Next, the holding strategy determination unit 307 determines that the next holding operation is "holding object 3 by sandwiching," which is the first holding operation in the holding strategy. Note that in the example of Table 1, the first score S of "sandwiching" for two or more objects is H0 When (I) has the same value, which one is given priority in the holding strategy can be determined appropriately by taking into consideration any factor such as the position or weight of object O.

[0096] In the example shown in Table 2, the second score T H,H0 The average value of subjects 1 to 5 in (I) is 0.79, which is less than 1. Therefore, it can be said that the ease of holding by the currently selected holding method H0 (clamping) is lower than the ease of holding by the holding method (suction) of the subject being evaluated. However, the second score T H,H0 Since (I) and its average value are both greater than the second threshold value Th2=0.5, the determining unit 306 of the control device 12 determines that it is not necessary to switch the holding method.

[0097] Next, as in the case of Table 1, the retention strategy determination unit 307 determines the order of retention based on the first score S H0 The holding strategy is determined so that the objects are held in the order of "object 5 → object 1 → object 2 → object 3 and object 4" using the currently selected holding method, "sandwiching." Next, the control device 12 stores the determined holding strategy in the memory unit 302. Next, the holding strategy determination unit 307 determines that the next holding operation is "holding object 5 by sandwiching," which is the earliest holding operation in the holding strategy.

[0098] In the example shown in Table 3, the second score T H,H0The average value of subjects 1 to 5 in (I) is 0.38, which is less than 1. Therefore, it can be said that the ease of holding by the currently selected holding method H0 (clamping) is lower than the ease of holding by the holding method (suction) of the subject being evaluated. In addition, the second score T H,H0 Since (I) and its average value are both below the second threshold value Th2=0.5, the determining unit 306 of the control device 12 determines that it is necessary to switch the holding method.

[0099] Next, the holding strategy determination unit 307 determines a first score S for each of the objects 1 to 5 based on the "holding" and "adsorption" strategies. H (I) is compared. In this example, the first score S of "adsorption" is obtained for all subjects 1 to 5. H (I) is the first score S of the currently selected holding method "Clip" H0 Since the retention strategy determination unit 307 determines the retention strategy to first switch the retention method, the retention strategy determination unit 307 determines the retention order after the switching of the retention method based on the first score S of the retention method after the switching, "adsorption". H (I) is set in descending order, and a holding strategy is determined so that holding is performed in the order of "target 1 and target 5 → target 2 → target 3 → target 4." Next, the control device 12 stores the determined holding strategy in the memory unit 302. Next, the holding strategy determination unit 307 determines that the next holding operation is "holding target 1 by adsorption" or "holding target 5 by adsorption," which is the earliest holding operation in the holding strategy. Here, target 1 and target 5 have a first score S of adsorption. H Since (I) is equal, which one should be given priority can be determined appropriately by taking into consideration any factors such as the positions and weights of objects 1 and 5.

[0100] Up to this point, in step S302, the first score S that is equal to or greater than the first threshold value Th1 H The processing flow of the control device 12 when it is determined that (I) exists has been described. In contrast, in step S302, if a first score S equal to or greater than the first threshold value Th1 is found, HIf it is determined that (I) does not exist (step S302: No), it is considered that there is no object O that is in a state that is sufficiently easy to hold, so instead of planning a holding strategy in steps S303 to S309, the control device 12 executes step S310 to plan a repositioning strategy that makes it easier to hold the object O.

[0101] In step S310, the holding impediment factor estimation unit 308 of the control device 12 estimates the presence or absence of various holding impediment factors that make it difficult to hold the object O in the source container V1. Examples of holding impediment factors include the objects O being crowded together in one part of the source container V1, the presence of an obstacle in front of the object O to be held, the object O to be held being positioned in an orientation that is not suitable for holding, the object O to be held having a shape that is not suitable for holding, etc.

[0102] The processing of the holding inhibition factor estimation unit 308 will be described using an example of a holding inhibition factor of densely packed objects O. The holding inhibition factor estimation unit 308 determines the degree of dense packing of objects O in the source container V1, for example, based on image data of the source container V1 acquired by the first sensor 11A. For example, the recognition processing unit 301 sets a mask area R2 for each object O in the image data acquired by the first sensor 11A, and calculates the center CO of the mask area R2 and the centroid FO of the object O (see FIG. 8). The recognition processing unit 301 also calculates the distance between the centers CO of the mask areas R2 and the distance between the centroids FO of the objects O as the distance between the objects O. The holding inhibition factor estimation unit 308 can estimate that the objects O are densely packed when the calculated distance between the objects O (a representative value such as an average value may be used) is equal to or less than a predetermined threshold. For example, the holding inhibition factor estimation unit 308 can estimate that the objects O are densely packed when one or more (e.g., multiple) center points CO, FO of another object O are included inside a circle with a radius of a predetermined threshold value from the center point CO, FO of a specific object O. By performing this density determination, the holding inhibition factor estimation unit 308 can determine whether or not there is a holding inhibition factor such as the objects O being densely packed.

[0103] In addition to or instead of the above method, the recognition processing unit 301 can determine whether the object O has an elongated shape (when viewed from above) based on the aspect ratio of the mask region R2 corresponding to the object O. For example, if it is determined that multiple nearby objects O have elongated shapes, the holding obstruction factor estimation unit 308 can estimate that thin objects O are placed vertically and crowded together. Alternatively, the holding obstruction factor estimation unit 308 may estimate that the object O is placed vertically and therefore is arranged in an orientation that is not suitable for holding.

[0104] In addition to the above, the holding impediment factor estimation unit 308 can estimate the presence or absence of various holding impediment factors. For example, based on the image data of the first sensor 11A, the holding impediment factor estimation unit 308 can estimate that there is an obstacle that prevents the holding of the object O to be held, or that the object O is positioned in an unsuitable posture for holding.

[0105] Next, the control device 12 executes step S311. In step S311, the placement change strategy determination unit 309 of the control device 12 determines a holding method and a placement change operation corresponding to the holding obstruction factor estimated in step S310. For example, if the presence of a holding obstruction factor, such as the objects O being crowded together, is estimated in step S310, the placement change strategy determination unit 309 can select, as the placement change operation, an operation of moving the objects O to eliminate the crowding. The placement change strategy determination unit 309 can select specific content of the placement change operation (e.g., an operation of pushing the object O, an operation of pulling the object O, an operation of pushing the object O away, an operation of scooping up the object O, etc.) depending on the density pattern. Furthermore, if the presence of a holding obstruction factor, such as an obstacle being located above the object O to be held, is estimated in step S310, the placement change strategy determination unit 309 can select, as the placement change operation, an operation of moving the obstacle. The association between such holding impediments and the rearrangement operation or the holding method to be used may be performed by referring to an association table or database, may be designated appropriately by the user, or may be learned by machine learning. However, at this stage, the object O to be subjected to the rearrangement operation has not yet been determined. Below, an explanation will be given with reference to Figs. 10 to 13, taking as an example a holding impediment that the objects O are crowded together. Figs. 10 to 13 are schematic diagrams showing an example of a rearrangement operation.

[0106] 10, in a case where five elongated objects O are crowded together in one corner of the source container V1, the rearrangement strategy determination unit 309 can select, as the rearrangement operation for the second object O from the left in the figure, a rearrangement operation of moving the object O in a direction from the centroid Go of the object O toward the centroid Gs of the area S where no object O is present. Here, the recognition processing unit 301 can set an area in the depth image D (see FIG. 6) that is deep (i.e., has a small height in the Z-axis direction of the world coordinate system) as the area S where no object O is present.

[0107] When setting the region S where no object O is present, the recognition processing unit 301 may search divided regions obtained by dividing the internal region of the source container V1 into multiple regions, instead of searching the entire internal region of the source container V1. For example, as shown in FIG. 11 , the recognition processing unit 301 may divide the entire source container V1 into two regions, upper and lower, along a dividing line indicated by a dashed dotted line, and search for a region where no object O is present in each of the upper and lower halves. For the upper half region, region S1 in FIG. 11 is set as the region where no object O is present, and the relocation strategy determination unit 309 may select a relocation operation in which the object O is moved in a direction from the centroid Go of the object O toward the centroid Gs1 of region S1. On the other hand, for the lower half region, region S2 in FIG. 11 is set as the region where no object O is present, and the relocation strategy determination unit 309 may select a relocation operation in which the object O is moved in a direction from the centroid Go of the object O toward the centroid Gs2 of region S2. By using this division process, the selection of the rearrangement operation can be adjusted as needed, for example, by limiting the direction of movement selected as the rearrangement operation to a specific desired area. Note that the method of dividing the area is not limited to the above example, and any other method can be used, such as dividing into two left and right or four parts.

[0108] The relocation strategy determination unit 309 may select not only the relocation operation but also the holding method depending on the estimated holding impediment. For example, if the relocation strategy determination unit 309 estimates that a holding impediment exists for which a relocation by "clamping" is effective, it may determine to perform the relocation using the first holding unit 200A. Alternatively, if the relocation strategy determination unit 309 estimates that a holding impediment exists for which a relocation by "suction" is effective, it may determine to perform the relocation using the second holding unit 200B. This allows the relocation to be performed efficiently using the holding means optimal for the determined relocation operation. However, the relocation strategy determination unit 309 may select the currently selected holding unit 200 as the holding method regardless of the estimated holding impediment and determine the relocation operation assuming the use of the currently selected holding unit 200. In this case, the number of times the holding means is switched is reduced, thereby improving the throughput of the transport system 1.

[0109] Generally, when attempting to hold objects O by "clamping," it is effective to change the arrangement to increase the distance between the objects O, whereas when attempting to hold objects O by "adsorption," it is effective to change the arrangement to increase the area of the area that can be adsorbed on the objects O. The operation of changing the arrangement of the objects O by "clamping" can be performed by inserting the tip of the clamping hand 202 between two objects O and opening the clamping hand 202 to increase the distance between the objects O, as shown in FIG. In addition, various other arrangement change operations can be selected, such as a method of inserting the tip of the clamping hand 202 between two objects O and then moving the clamping hand 202 in one direction or rotating the base axis (six axes in the case of an articulated robot) of the clamping hand 202 to increase the distance between the objects O, a method of moving the clamping hand 202 sideways while keeping the tip of the clamping hand 202 in contact with the top surface of the object O to tip the object O, a method of slightly lifting a specific object O with the clamping hand 202 and then dropping it to change the arrangement, a method of lifting a specific object O with the clamping hand 202 and dropping it on top of other objects O, and a method of moving the specific object O while clamping it so as to knock down other objects O (for example, dragging the object O while holding the edge of the object O). Furthermore, the clamping hand 202 can smooth out the uneven surface of the object O before gripping it, or can deform the object O by pressing the object O against a wall to grip it.

[0110] The operation of changing the position of the object O by "adsorption" can be performed by, for example, moving the suction device 203 sideways while the suction unit 205 of the suction device 203 is in contact with the top surface of the object O, and tipping the object O sideways, as shown in Fig. 13. In addition, various other position change operations can be selected, such as a method of rotating the suction device 203 while the suction unit 205 is in contact with the object O, a method of lifting a specific object O with the suction device 203 and dropping it onto other objects O, and a method of moving a specific object O while adsorbing it so as to knock down other objects O.

[0111] Next, the control device 12 executes step S312. In step S312, the score calculation unit 304 calculates a "third score" as an index of the effectiveness of the rearrangement operation. The score calculation unit 304 calculates a third score for each object O or a region of the object O based on the holding inhibition factor estimated in step S310 and the holding method and rearrangement operation determined in step S311. For example, if a holding inhibition factor of dense objects O is estimated in step S310, and a rearrangement operation of "moving one object O toward a region S where no object O exists" is determined to be performed with "clamping" selected as the holding method in step S311, the score calculation unit 304 calculates a third score representing the effectiveness of such a rearrangement operation for each object O or a region of the object O that can be a holding target. By comparing these third scores, it is possible to select an object O or a region of the object O that is preferable as a target for the rearrangement operation.

[0112] The score calculation unit 304 can calculate the third score based on, for example, the ease of rearranging the target object O, the difficulty of holding the target object O, and the number of other objects O expected to be rearranged as a result of the rearrangement operation of the target object O. The easier the rearrangement operation of the target object O is, the higher the effectiveness of the rearrangement operation can be evaluated. The more difficult the target object O is to hold, the easier it is to hold it through the rearrangement operation, and therefore the higher the effectiveness of the rearrangement operation can be evaluated. The relationship between the number of objects O rearranged at once by a single rearrangement operation and the effectiveness of the rearrangement operation may vary depending on factors such as the size of the objects O. In other words, there is an appropriate number for the number of objects O rearranged at once by a single rearrangement operation, depending on factors such as the size of the objects O, and the closer the number is to that number, the higher the effectiveness of the rearrangement operation can be evaluated. For example, when the objects O are small, the more objects O are rearranged at once, the easier it is to hold them, and the higher the effectiveness of the rearrangement operation. On the other hand, when the objects O are large, if too many objects O are rearranged at once, it may become more difficult to hold them.

[0113] The ease of the rearrangement operation of the object O can be evaluated in the same manner as the estimation of the holding inhibition factors in step S310, for example, based on the arrangement and shape of the object O, the degree of crowding of the objects O, etc. The difficulty of holding the object O can be evaluated, for example, based on the same evaluation items as the first score indicating the ease of holding, so as to have a negative correlation with the first score. The number of other objects O expected to be rearranged in conjunction with the rearrangement operation of the target object O can be evaluated, for example, based on the arrangement of each object O and the content of the rearrangement operation.

[0114] The score calculation unit 304 can calculate the third score using an evaluation function expressed as a linear combination of the evaluation items. However, the evaluation function is not limited to the above example, and any evaluation function such as a nonlinear function can be used. Furthermore, the evaluation items are not limited to the above example, and may be, for example, quantities dependent on the characteristics of the object O, the characteristics of the holding unit 200 used, the characteristics of the selected rearrangement operation, etc.

[0115] Next, the control device 12 executes step S313. In step S313, the placement change strategy determination unit 309 of the control device 12 determines a placement change strategy based on the calculated third score. For example, the placement change strategy determination unit 309 may determine to execute the placement change operation determined in step S311 on the object O or the region of the object O having the largest third score. Note that, similar to the retention strategy, the placement change strategy determination unit 309 may sort the objects O or the regions of the object O in descending order of third score to determine the order of placement changes.

[0116] Next, the control device 12 executes step S314. In step S314, the control device 12 stores the rearrangement strategy determined by the rearrangement strategy determination unit 309 in step S313 in the storage unit 302. In this way, the control device 12 can create a rearrangement strategy for the object O when it is determined that it is difficult to hold the object O.

[0117] In the above example, the third score is calculated in step S312 based on the holding method and arrangement change operation determined in step S311, but the calculation of the third score is not limited to this. For example, in step S312, the score calculation unit 304 can also calculate the third score for each object O or region of the object O for multiple holding methods and / or multiple change operations. For example, if there are 10 objects O or regions of the object O to be subjected to score calculation, and the third score is calculated for each of the two holding methods, "clamp" and "suck," a total of 10 × 2 = 20 third scores are calculated. Furthermore, for example, if five types of rearrangement operations are predefined, such as (1) "inserting the tip of the holding unit 200 between the objects O and opening it," (2) "inserting the tip of the holding unit 200 between the objects O and moving it in one direction," (3) "inserting the tip of the holding unit 200 between the objects O and rotating it," (4) "moving the tip of the holding unit 200 laterally while in contact with the top surface of the object O," and (5) "lifting and dropping the object O," the control device 12 can select one or more of the above rearrangement operations as needed to calculate the score based on various information acquired by the sensor 11, and calculate a third score for each selected rearrangement operation. By selecting the rearrangement operation to be used for score calculation according to the situation, the calculation time and load can be reduced compared to performing all of the defined types of rearrangement operations. For example, if there are 10 objects O or regions of the object O to be calculated, and two types of rearrangement operations are selected as candidates, a total of 10 × 2 = 20 third scores are calculated. By selecting the largest third score among the third scores calculated in this manner, it is possible to determine not only the object O to be held or the area of the object O, but also the holding method and placement change operation to be adopted based on the third score. In this case, steps S310 and S311 may be omitted. Note that the control device 12 may omit the selection of the placement change operation and calculate the third score for all types of placement change operations.

[0118] As an example of a method for selecting an appropriate rearrangement operation as described above, the control device 12 can use, for example, an evaluation function g(I) shown in the following Equation 6. g(I) represents a third score of the rearrangement operation for the object O or the mask region I indicating the region of the object O.

number

[0119] In Equation 6, f1 is the flexibility of object O. f2 is the surface area of the portion of object O that is expected not to overlap with other objects O (i.e., be exposed) after the rearrangement. f3 is the aspect ratio of object O. f4 is the difference in height between object O and its surroundings (inner and outer height). f5 is the spatial margin that represents the size of the space in the surrounding area of object O that is free from other objects O, etc. w1 to w5 represent the weights of each function.

[0120] The relationship between the parameters f1 to f5 of the evaluation function g(I) and the above-mentioned rearrangement operations (1) to (5) can be expressed, for example, as shown in the table below. For example, when the f1 component is the largest in the evaluation function g(I), (1) and (3) are obtained as candidates for the rearrangement operation. In this case, since there are multiple candidates, the control device 12 checks the parameter that is the next largest after f1. Specifically, the control device 12 compares the parameters f3 and f4 corresponding to (1) or (3), and selects the rearrangement operation (3) if the f3 component is larger than the f4 component in the evaluation function g(I), or selects the rearrangement operation (1) if the f3 component is smaller than the f4 component. Note that the control device 12 may omit the process of comparing f3 and f4, calculate third scores for both the rearrangement operations (1) and (3), and select the rearrangement operation with the larger third score. [Table 4]

[0121] The control device 12 may execute the above-described selection process of the rearrangement operation instead of steps S310 and S311. In this case, the control device 12 can select an appropriate type of rearrangement operation without specifically estimating the retention inhibiting factor, thereby reducing the calculation cost.

[0122] 3 again, next, the control device 12 executes step S4. In step S4, the determination unit 306 of the control device 12 determines whether or not a rearrangement is necessary. If it is determined that no first score equal to or greater than the first threshold value Th1 exists (step S302: No), the determination unit 306 determines that a rearrangement is necessary, and if it is determined that a first score equal to or greater than the first threshold value Th1 exists (step S302: Yes), the determination unit 306 determines that a rearrangement is not necessary. If the determination unit 306 determines that a rearrangement is necessary (step S4: Yes), the control device 12 executes step S5.

[0123] In step S5, the operation control unit 303 of the control device 12 specifically calculates the position where the relocation operation of the target object O is to be performed and the posture of the movable arm 100 for performing the relocation operation, based on the relocation strategy determined in step S313. Based on the calculation result, the operation control unit 303 controls the movable arm 100 and the holder 200 to relocate the object O (relocation process). That is, the relocation operation is performed on the object O with the largest third score so that the holder 200 can easily hold the object O. Thereafter, the process of the control device 12 returns to step S2, and the first sensor 11A acquires image data of the state of the source container V1 after the relocation. Next, in step S301, the first scores are calculated again, and in step S302, each first score is compared with the first threshold Th1. In this manner, the relocation operation is repeated until the first scores of one or more objects O become equal to or greater than the first threshold Th1.

[0124] On the other hand, if the determination unit 306 determines that a change in the arrangement is not necessary (step S4: No), the control device 12 executes step S6. In step S6, the operation control unit 303 of the control device 12 specifically calculates the position at which the target object O determined in step S3 will be held and the posture of the movable arm 100 when holding the target object O (holding position and posture planning step).

[0125] If the holding method for object O selected in step S3 is "clamping," in step S6, the operation control unit 303 of the control device 12 specifically calculates the position for clamping object O and the posture of the movable arm 100 when clamping, using a method appropriately selected from known methods.

[0126] If the holding method for object O selected in step S3 is "adsorption," in step S6, the operation control unit 303 of the control device 12 specifically calculates the position at which object O will be adsorbed and the posture of the movable arm 100 when adsorbing, using a method appropriately selected from known methods.

[0127] In this embodiment, the selectable holding methods are either "clamping" or "adsorption," but if, for example, "clamping and adsorption (hybrid)" is available, in step S6, the operation control unit 303 of the control device 12 can specifically calculate the position at which the object O is clamped and adsorbed and the posture of the movable arm 100 when clamping and adsorbing, using a method appropriately selected from known methods.

[0128] The operation control unit 303 determines the position where the object O is held and the manner in which the object O is held. Movable arm After calculating 100 postures, it is calculated whether the calculated holding motion is possible as an actual robot motion, and if it is determined that the actual motion is impossible, the position to hold object O and the motion to hold it are calculated again. Movable arm 100 postures are calculated. If it is determined that the actual movement is impossible, the process may return to step S3 and start again from planning the holding strategy.

[0129] Calculating the position where object O is held and the posture of the movable arm 100 requires a very large amount of calculation. The control device 12 only needs to calculate the position where object O is held and the posture of the movable arm 100 for the selected object O. Therefore, the transport system 1 can significantly reduce the amount of calculation required compared to other transport systems that calculate the position where object O is held and the posture of the movable arm 100 in order to select the object O to be held.

[0130] Next, the control device 12 executes step S7. In step S7, the determination unit 306 of the control device 12 determines whether or not switching of the holding method is necessary. If the determination unit 306 determines that switching of the holding method is necessary (step S7: Yes), that is, if the currently selected holding method differs from the holding method for the next holding operation, the control device 12 executes step S8. In step S8, the operation control unit 303 switches the holding method (holding method switching process). For example, when switching from "clamping" to "suction," the operation control unit 303 controls the movable arm 100 of the handling device 10 so that the second holding unit 200B faces the source container V1 instead of the first holding unit 200A, which faces the source container V1. On the other hand, if the determination unit 306 determines that switching of the holding method is not necessary (step S7: No), that is, if the currently selected holding method and the holding method for the next holding operation are the same, the control device 12 proceeds to step S9 without executing step S8.

[0131] Next, the control device 12 executes step S9. In step S9, the control device 12 controls the holding unit 200 and the movable arm 100 based on the position for holding the object O and the posture of the movable arm 100 calculated in step S6 (operation control step). The selected object O is transported by the handling device 10 from the source container V1 to the destination container V2.

[0132] Next, the control device 12 executes step S10. In step S10, the control device 12 determines whether the holding and / or transport of the target object O has been successful based on, for example, image data acquired by the first sensor 11A and the second sensor 11B (holding success / failure determination step). The control device 12 stores the determination result in the memory unit 302.

[0133] Next, the control device 12 executes step S11. In step S11, the determination unit 306 of the control device 12 determines whether the orders listed in the order list have been completed. For example, the determination unit 306 determines whether an object O remains in the source container V1. If the determination unit 306 determines that an object O remains in the source container V1 (step S11: No), the control device 12 executes step S2 again. That is, the control device 12 again acquires information regarding the state of the source container V1 after picking up the object O, the state of each holding unit 200A, 200B, etc., and creates a next holding strategy based on this information. Note that the control device 12 may determine the next holding operation based on a holding strategy including the holding order that has already been created, without performing the holding strategy planning process. Alternatively, the control device 12 may modify or update the already created holding strategy based on the acquired information, instead of performing the same holding strategy planning process. In addition, after performing one or more holding operations, the control device 12 may cause the score calculation unit 304 to change the value of the parameter used when generating the score, or may cause the threshold generation unit 305 to change the first threshold Th1 and / or the second threshold Th2.

[0134] On the other hand, if the determination unit 306 determines that the order written in the order list has been completed (step S11: Yes), the control device 12 11 For example, if the determination unit 306 determines that there is no object O remaining in the source container V1, the control device 12 executes step S 11 Execute and end control.

[0135] According to the configuration of the first embodiment described above, when an object is difficult to hold, the ease of holding brought about by the rearrangement operation can be improved by evaluating the effectiveness of the rearrangement operation for each object or object region that can be subject to rearrangement, thereby shortening the operating time of the handling device and improving the throughput of the system.

[0136] Furthermore, in this embodiment, the control device 12 calculates a score (third score) of the rearrangement operation for each object O in the evaluation, and selects the object O to be rearranged based on the score. The score of the rearrangement operation calculated for each object O is calculated based on at least one of the ease of the rearrangement operation of the target object O, the difficulty of holding the target object O, and the number of objects O expected to be rearranged as a result of the rearrangement operation of the target object O. This makes it possible to select the object O for which the rearrangement operation is expected to be most effective as the target of the rearrangement operation.

[0137] Furthermore, in this embodiment, the control device 12 estimates the holding impediment factors that prevent the holding of the object O based on the information acquired from the sensor 11, and determines the placement change operation based on the estimated holding impediment factors and the evaluation results. This allows the placement change method to be determined in accordance with the factors that are currently preventing the holding operation, thereby improving the effectiveness of the placement change operation.

[0138] Furthermore, in this embodiment, the control device 12 determines the degree of crowding of the objects O based on at least one of the arrangement and shape of the objects O, and determines a rearrangement operation to reduce the degree of crowding of the objects O. This allows for rearrangement to be performed to eliminate the crowding of the objects O when it is difficult to hold the objects O due to their crowding.

[0139] Furthermore, in this embodiment, the holder 200 is configured to hold the object O by at least one of clamping and suction, and when the holder 200 holds the object O by clamping, the control device 12 determines the rearrangement operation to increase the distance between the objects O, and when the holder 200 holds the object O by suction, the control device 12 determines the rearrangement operation to increase the area available for suction of the object O. This makes it possible to select an appropriate rearrangement operation depending on the type of holding means.

[0140] Furthermore, in this embodiment, the holding unit 200 can select one or more holding methods from a plurality of holding methods to hold the object O, and the control device 12 determines the object O to be subjected to the rearrangement operation and the holding method of the holding unit 200 to be used for the rearrangement operation based on the evaluation results. This makes it possible to select a holding means that makes it easy to perform an effective rearrangement operation.

[0141] Furthermore, in this embodiment, the holding unit 200 can select one or more of a plurality of holding methods to hold the object O, and the holding method currently selected by the holding unit 200 is used as the holding method of the holding unit 200 to be used in the placement change operation. This makes it possible to avoid changing the currently selected holding method, and improves the throughput of the system.

[0142] Furthermore, in this embodiment, the control device 12 calculates an ease-of-holding score (first score) for each object O based on information acquired from the sensor 11, and determines whether or not to change the placement of the object O based on the ease-of-holding score. The control device 12 determines to change the placement of the object O when there is no object O whose ease-of-holding score is equal to or greater than a predetermined threshold (first threshold). As a result, when there is an object O that is recognized to have a certain degree of ease of holding, the holding operation is prioritized over the placement change operation, and the throughput of the system can be improved.

[0143] Furthermore, in this embodiment, the control device 12 selects one or more types of rearrangement operations from among the multiple types of rearrangement operations based on at least one of information about the object O and information acquired from the sensor 11, and evaluates the effectiveness of the selected types of rearrangement operations for each object O. The control device 12 selects one or more types of rearrangement operations from among the multiple types of rearrangement operations based on one or more pieces of information selected from the group consisting of the flexibility of the object O, the surface area of the portion of the object O that is expected to be exposed after the rearrangement operation, the aspect ratio of the object O, the height difference between the object O and its surroundings, and the size of the space in the area surrounding the object O. This makes it possible to significantly reduce calculation costs compared to calculating the third score for each object O for all types of rearrangement operations.

[0144] Furthermore, in this embodiment, the control device 12 calculates the position where the determined rearrangement operation is to be performed and the posture of the movable arm 100 for performing the determined rearrangement operation. This allows the control device 12 to create a rearrangement strategy without specifically calculating the position where the rearrangement operation is to be performed or the posture of the movable arm 100. The control device 12 only needs to calculate the position where the rearrangement operation is to be performed and the posture of the movable arm 100 for only the object O finally selected as the object to be rearranged, thereby significantly reducing the amount of calculation required.

[0145] In this embodiment, the handling device 10 includes a movable arm 100, a holding unit 200, a sensor 11, and a control device 12. The holding unit 200 is attached to the movable arm 100 and is capable of holding an object O by selecting one or more holding methods from a plurality of holding methods. The sensor 11 is capable of detecting a plurality of objects O. The control device 12 controls the movable arm 10 0Based on the information acquired from the sensor 11, the control device 12 calculates a score (first score) for each object O and each holding method, based on the selected holding method. The control device 12 selects the next object O to be held and the holding method based on the score. The control device 12 calculates the position at which the selected object O will be held and the posture of the movable arm 100. This makes it possible to efficiently determine a holding strategy for holding the object O while reducing the number of times the holding method is switched. The control device 12 selects a holding strategy without specifically calculating the position at which the object O will be held or the posture of the movable arm 100. A holding strategy is, for example, selecting the type of handling device 10 to be used, the object O to be held, and the holding method. The control device 12 only needs to calculate the position at which the object O will be held and the posture of the movable arm 100 for the selected object O, which significantly reduces the amount of calculation required. In addition, if the currently selected retention method is relatively easy to maintain, a retention strategy can be created that prioritizes the use of the currently selected retention method without switching retention methods, thereby saving the time required to switch retention methods and thereby shortening the overall work time.

[0146] In this embodiment, the plurality of holding methods include clamping and suction, which allows the handling device 10 to hold various objects O, such as thin objects O that are difficult to hold by clamping and objects O with uneven shapes that are difficult to hold by suction.

[0147] Furthermore, in this embodiment, the control device 12 selects the object O to be held and the holding method based on the score so as to reduce the number of times the holding method is switched. The control device 12 also determines whether or not it is necessary to switch the holding method based on the score. This allows the handling device 10 to perform a series of holding operations so as to minimize the number of times the holding method is switched, thereby reducing the time required for the entire work.

[0148] In this embodiment, the control device 12 determines whether or not it is necessary to switch the retention method based on the magnitude relationship between the score and a predetermined second threshold value Th2. This allows the control device 12 to easily determine the need to switch the retention method. Furthermore, the control device 12 can create a retention strategy according to the situation by adjusting the second threshold value Th2.

[0149] Furthermore, in this embodiment, the control device 12 determines the order in which the object O is to be held based on the score. This allows the control device 12 to select the next holding operation in consideration of the entire pickup operation specified in the order list.

[0150] In this embodiment, the control device 12 determines the order in which to switch the object holding method, thereby enabling the control device 12 to select the optimal timing for switching the holding method in consideration of the entire pickup operation specified in the order list.

[0151] Furthermore, in this embodiment, when calculating the score when the holding method is suction, the control device 12 calculates the score for each region of the object O. This allows the control device 12 to select the region of the same object O that is easiest to hold as the holding target region, thereby improving holding efficiency.

[0152] Furthermore, in this embodiment, the control device 12 calculates the ease of holding for each object O and for each holding method as a first score, and the score based on the selected holding method is a second score calculated based on the first score for each object O and for each holding method. This allows the control device 12 to determine the holding operation based on two perspectives: the ease of holding the object O represented by the first score, and the priority based on the currently selected holding method represented by the second score.

[0153] In this embodiment, the second score for the object O and the holding method is calculated by multiplying the first score for the object O or the region of the object O and the selected holding method by the first score for the object O or the region of the object O and the The second score is evaluated This is a value divided by the first score for the holding method, which allows the control device 12 to easily calculate the second score for determining the need to switch the holding method.

[0154] (Second embodiment) Next, a second embodiment will be described with reference to Fig. 14. The second embodiment differs from the first embodiment in that score calculation is corrected depending on whether or not retention is successful. Note that the configuration other than that described below is the same as that of the first embodiment. FIG. 14 is a control flowchart of a retention strategy planning process by the control device 12 according to the second embodiment.

[0155] In the second embodiment, the control device 12 executes steps S11 and S12 after step S10 (holding success / failure determination process) of the first embodiment. As in the first embodiment, the control device 12 plans a holding strategy or a rearrangement strategy based on the order list and information acquired by the sensor 11, controls the movable arm 100 and the holding unit 200 to hold or rearrange the object O based on the planned strategy, and if holding is performed, determines in step S10 whether the object O has been successfully held.

[0156] After step S10, if the determination unit 306 determines that the holding was successful (step S11: Yes), it determines whether the order list has been completed, as in the first embodiment (step S13). On the other hand, if the determination unit 306 determines that the holding was unsuccessful (step S11: No), the control device 12 executes step S12. In step S12, the control device 12 identifies, as a holding failure area, an area in which the holding failed object O is located, for example, by referring to the holding strategy stored in the memory unit 302 in step S308. The score calculation unit 304 performs correction so that the first score of the object O included in the identified holding failure area becomes smaller. Note that instead of or in addition to the above example, the threshold generation unit 305 may correct the first threshold Th1.

[0157] The hold failure area may be set, for example, to include only the object O that failed to be held, or may be set as an area of any shape, such as a circle or polygon, having a predetermined size centered on the object O that failed to be held. Alternatively, the internal area of the source container V1 may be divided into multiple areas in advance, and one of the divided areas that includes the object O that failed to be held may be set as the hold failure area. However, the method for setting the hold failure area is not limited to the above example, and any method may be employed.

[0158] The correction process by the score calculation unit 304 may be performed, for example, by multiplying the first score calculated normally by a correction coefficient greater than or equal to 0 and less than 1 when the first score of the object O included in the retention failure region is next calculated, or by correcting the parameters of the evaluation function used to calculate the score of the object O included in the retention failure region. However, the method of correcting the first score is not limited to the above example, and any method can be adopted. Note that the correction process may be performed in another step, such as calculating the second score instead of the first score, so that the priority of the object O in the retention strategy is corrected.

[0159] Next, the control device 12 determines whether the order list has been completed as in the first embodiment (step S13).

[0160] According to the configuration of this embodiment, the control device 12 calculates a score (first score or second score) for determining the priority of the holding operation for each object O based on information acquired from the sensor 11, and when it is determined that the holding operation by the holding unit 200 has failed, corrects the score for determining the priority based on information about the object O for which the holding operation failed. This allows the control device 12 to appropriately correct the priority of each object O in the holding strategy depending on whether the holding operation was successful or not. As a result, it is possible to avoid a situation in which the holding unit 200 repeatedly attempts a holding operation on an object O that is difficult to hold, thereby improving the success rate of the holding operation in the holding strategy.

[0161] (Third embodiment) Next, a third embodiment will be described with reference to Fig. 15. The third embodiment differs from the first embodiment in that a plurality of retention strategies are created. Note that the configuration other than that described below is the same as that of the first embodiment. FIG. 15 is a control flowchart of a retention strategy planning process by the control device 12 according to the third embodiment.

[0162] In the third embodiment, the control device 12 executes steps S357 to S359 instead of step S307 in the first embodiment. In the retention strategy planning step (step S3), the control device 12 calculates the first score S H (I) is calculated (step S351), and a first score S that is equal to or greater than the first threshold value Th1 is calculated. H If (I) does not exist (step S352: No), similarly to the first embodiment, a holding inhibiting factor is estimated (step S362), a holding method and a rearrangement operation are determined (step S363), a third score is calculated for each object O or region of the object O (step S364), a rearrangement strategy is determined (step S365), and the rearrangement strategy is stored in the storage unit 302 (step S366). H If (I) exists (step S352: Yes), the control device 12 calculates the second score T H,H0(I) (step S353), and determines whether switching of the holding method is necessary (step S354). If the determination unit 306 determines that switching of the holding method is unnecessary (S355: No), the holding strategy determination unit 307 of the control device 12 determines a holding strategy including the order of holding (step S356), stores the holding strategy in the storage unit 302 (step S360), and determines the next holding operation based on the holding strategy (step S361), as in the first embodiment. In other words, the control content of the control device 12 when the determination unit 306 determines that switching of the holding method is unnecessary is the same as in the first embodiment.

[0163] On the other hand, if the determination unit 306 determines that the retention method needs to be switched (S355: Yes), the control device 12 executes step S357. In step S357, the retention strategy determination unit 307 of the control device 12 creates a plurality of retention strategy proposals including the order of retention and switching. For example, the retention strategy determination unit 307 determines the first score S H Based on (I), a retention strategy proposal with one switch and a retention strategy proposal with a relatively low overall success rate and a relatively high overall success rate and a retention strategy proposal with two switches are created.

[0164] Next, the control device 12 executes step S358. In step S358, the score calculation unit 304 of the control device 12 calculates the score of each proposed retention strategy created by the retention strategy determination unit 307. For example, the score calculation unit 304 calculates the number of times the retention method is switched in the proposed retention strategy, the first score S for each retention operation, H The score of each proposed retention strategy is calculated using evaluation items such as the average, maximum, and minimum values of (I), the estimated time required to complete the retention strategy, etc. The score calculation unit 304 can also change the weight of the evaluation items as appropriate, for example, in response to input by the operator.

[0165] Next, the control device 12 executes step S359. In step S359, the retention strategy determination unit 307 of the control device 12 determines an optimal retention strategy based on the score of each retention strategy proposal generated by the score calculation unit 304. For example, the retention strategy determination unit 307 selects the retention strategy proposal with the highest score as the retention strategy.

[0166] Thereafter, the control device 12 stores the holding strategy determined by the holding strategy determination unit 307 in the storage unit 302 (step S360), and determines the next holding operation based on the holding strategy (step S361).

[0167] The control device 12 may create multiple retention strategy proposals based on the first score, omitting the calculation of the second score and / or the determination of the need for switching (steps S353 to S355). For example, the retention strategy determination unit 307 can create a retention strategy proposal that does not involve switching and has the lowest overall success rate, a retention strategy proposal that involves switching once and has the next lowest overall success rate, and a retention strategy proposal that involves switching twice and has the highest overall success rate. The score calculation unit 304 calculates a score for each retention strategy proposal, and the retention strategy determination unit 307 determines a retention strategy based on these scores. This allows the control device 12 to determine a retention strategy that reduces the number of times the retention method is switched, without having to determine whether to switch the retention method.

[0168] According to the configuration of this embodiment, the control device 12 creates a plurality of holding strategies including at least one of the order in which the object O is held and the order in which the holding method is switched, and selects one of the plurality of holding strategies. This allows the control device 12 to compare and consider a wide range of holding strategies and select the holding strategy that is best in terms of the success rate, the number of times the holding method is switched, etc.

[0169] In this embodiment, the control device 12 selects one of the holding strategies so as to minimize the number of times the holding method is switched, thereby enabling the handling device 10 to execute a series of holding operations so as to minimize the number of times the holding method is switched, thereby reducing the time required for the entire operation.

[0170] (Fourth embodiment) Next, a fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that a holding method is selected based on the frequency of use of each holding method. Note that the configuration other than that described below is the same as that of the first embodiment.

[0171] The control device 12 records the frequency of use of each holding method in the storage unit 302. For example, the control device 12 can record the holding method used in the storage unit 302 every time a holding operation is performed, and calculate the frequency of use of each holding method based on the history of holding method use. Alternatively, the control device 12 can record in the storage unit 302 that the holding method has been switched, and calculate the frequency of use of each holding method based on the history of holding method switching. This allows the control device 12 to check how frequently each holding method has been used in the past.

[0172] When there is a bias in the calculated frequency of use of each holding method or when it is determined that the limit of use of a holding method is approaching, the control device 12 can reduce the frequency of use of a holding method that is frequently used. For example, the control device 12 calculates the second score T H,H0 When calculating (I), a second score T is used to lower the priority of frequently used retention methods. H,H0 Alternatively, the control device 12 may adjust the parameters in (I). Alternatively, the control device 12 may adjust the second score T H,H0When comparing (I) with the second threshold value Th2, the threshold value generating unit 305 can be made to adjust the second threshold value Th2 so as to lower the priority of selecting a frequently used holding method. Alternatively, for example, when determining a holding strategy including the order of holding and switching in step S307 of FIG. 9, the control device 12 can adjust the first score S by another holding method so as to avoid using a frequently used holding method as much as possible. H For an object O with a relatively high (I), the control device 12 can be configured to hold the object O using the other holding method. Alternatively, when calculating the score of each proposed holding strategy in step S358 of Fig. 15, the control device 12 can adjust a parameter in the score so as to lower the priority of selection of a holding method that is frequently used.

[0173] For example, the controller 12 may use the following equation: 7 The scores for clamping and suction can be calculated by:

number

[0174] Formula 7 In this case, w1h1+w2h2+w3=1, and w i ≧0(i=1,2,3), h j ≧0(j=1,2). S p (I) is the first score when the object O or the mask area I indicating the area of the object O is held by "pinching" it. S s (I) is the first score when the object O or the mask area I representing the area of the object O is held by "adsorption." g(S H (I),S H0 (I)) is a function for calculating the second score. item is a value normalized by multiplying the weight of the object O by a coefficient. h1 and h2 are weights that adjust the degree of use of the current holding method. For example, the weights of h1 and h2 can be stored in a database or the like as table information as shown below. In addition, the control device 12 calculates the first score S p (I) or Ss The weights may be adjusted based on the calculation results of (I). [Table 5]

[0175] According to the configuration of the fourth embodiment as described above, the control device 12 selects the object O to be held and the holding method based on the frequency of use of each holding method. This allows the control device 12 to suppress bias towards a specific holding method by making adjustments such as lowering the priority of holding methods that are used frequently, thereby extending the life of the holding unit 200.

[0176] In this embodiment, the control device 12 calculates a score based on the frequency of use of each holding method. Alternatively, the control device 12 determines a second threshold value Th2 based on the frequency of use of each holding method, and determines whether or not it is necessary to switch the holding method based on the magnitude relationship between the score and the second threshold value Th2. This allows the control device 12 to easily make adjustments such as lowering the priority of frequently used holding methods by correcting the score or threshold value based on the frequency of use of each holding method.

[0177] (Fifth embodiment) Next, a fifth embodiment will be described. The fifth embodiment differs from the first embodiment in that a holding method is selected based on the detection result of the physical state of each holding unit 200A, 200B. Note that the configuration other than that described below is the same as that of the first embodiment.

[0178] The control device 12 receives information about the physical conditions of the first and second holding units 200A and 200B from the third and fourth sensors 11C and 11D. Repeated use of a particular holding unit 200 can cause changes in the measurements of the physical sensors of the third and fourth sensors 11C and 11D due to distortion, changes in shape, changes in the surface condition, and so on of the holding unit 200. Based on this information about the physical conditions, the control device 12 estimates the usage conditions, such as the wear state, of the holding units 200A and 200B. As with the fourth embodiment, the control device 12 adjusts the score, threshold, order of holding and switching in the holding strategy, and so on, to lower the priority or frequency of selection of a holding method using the holding units 200A and 200B estimated to be deteriorated or damaged.

[0179] According to the configuration of the fifth embodiment as described above, the control device 12 selects the object O to be held and the holding method based on physical information of the holding unit 200. This makes it possible to adjust the priority of the holding method while actually measuring physical deterioration of the holding unit 200, thereby extending the life of the holding unit 200.

[0180] (Sixth embodiment) Next, a sixth embodiment will be described. The sixth embodiment differs from the first embodiment in that an algorithm using machine learning is used instead of a rule-based algorithm. Note that the configuration other than that described below is the same as that of the first embodiment.

[0181] In this embodiment, the calculation of the first score, second score, and third score in the first embodiment may be performed by machine learning. For example, when supervised learning is assumed, the evaluation function defined above may be used as an evaluation value during learning. The learning algorithm is not limited to supervised learning, and may be changed depending on the type of learning, such as unsupervised learning or reinforcement learning.

[0182] Regarding the switching of the holding method, the control device 12 configures a learning network based on the information acquired from the sensor 11, the information obtained by the first score, the information obtained by the second score, and the information required for the number of times the holding method is switched, thereby learning to evaluate the success rate of the holding action, the number of times the holding method is switched, and the frequency of use of the holding method, and adjusting the balance between these. For example, the control device 12 trained by machine learning can perform the information acquisition process by the sensor 11 (step S 2 ) as input, and can output scores for each retention target and retention method, one or more retention strategies including the order of retention and switching, suitable for increasing the success rate of retention actions, suppressing switching of retention methods, and suppressing bias in the frequency of use of retention methods.

[0183] When using machine learning, in the evaluation function of the first score of Equation 2 and Equation 3, w p1 =w p2 =w p3 =w p4 =w p5 = 0, and w s1 =w s2 =w s3 =w s4 =w s5 =0.

[0184] Furthermore, with regard to the rearrangement operation, the control device 12 can perform learning to optimize the determination of whether or not to perform a rearrangement or to improve the effectiveness of the rearrangement operation by configuring a learning network based on the information acquired from the sensor 11, the information obtained by the third score, the success or failure of retention, etc. For example, the control device 12 trained by machine learning can perform learning to optimize the determination of whether or not to perform a rearrangement or to improve the effectiveness of the rearrangement operation by performing the information acquisition process by the sensor 11 (step S 2 ) as input, and can output a third score, a re-arrangement strategy, etc. that are suitable for improving the appropriateness of the decision on whether to re-arrange or not, and the selection of effective re-arrangement operations, thereby improving the throughput of the system.

[0185] The learning network for switching the holding method and the learning network for the arrangement change operation may be integrated into one.

[0186] According to the configuration of the sixth embodiment as described above, the control device 12 can efficiently output more appropriate scores, retention strategies, and allocation change strategies by repeatedly performing machine learning.

[0187] In each of the above embodiments, the control device 12 calculates the second score T H,H0 The order of the holding operation and the switching operation is determined after determining the necessity of switching by comparing (I) with the second threshold value Th2, but the order of the holding operation and the switching operation may be determined so as to reduce the number of switching operations without determining the necessity of switching. For example, the score calculation unit 304 of the control device 12 calculates a first score S that indicates the ease of holding simply without considering the currently selected holding method. H For (I), the first score S by the currently selected retention method H0 H0 Correct the value of only (I) (for example, multiply by n (n>1)), and the first score S H (I) is the second score T H,H0 (I) is generated and this second score T H,H0 Based on (I), a holding strategy including the order of holding operations and switching operations may be determined so as to minimize the number of switching operations. In this case, the control device 12 does not need to determine the necessity of switching the holding method. H (I) Second score T including the correction method H,H0 The generation method of (I) is not limited to the above example, and any method can be used as long as it has some bias to preferentially use the currently selected retention method.

[0188] In the above embodiments, the handling device 10 includes a first handling device 10A that performs "clamping" and a second handling device 10B that performs "suction." However, the configuration of the handling device 10 is not limited to this. The handling device 10 may include one or more holding units that perform any holding method as long as it can select one or more of a plurality of holding methods to hold an object. For example, the handling device 10 may include another handling device in addition to the first handling device 10A and the second handling device 10B. The handling device 10 may include a single hybrid hand that can perform both "clamping" and "suction" instead of the first handling device 10A and the second handling device 10B. Furthermore, at least one of the first handling device 10A and the second handling device 10B may be a hybrid hand that can perform both "clamping" and "suction." Such a hybrid hand may be configured, for example, to switch between a clamping unit and a suction unit using a rotating unit that rotates 180 degrees, or may have a revolver-type configuration like an optical microscope, or may have other configurations. The handling device 10 may have two types of suction handling devices that differ in size, shape, characteristics, etc. Alternatively, the handling device 10 may have a handling device that holds the object O by a holding method other than clamping and suction.

[0189] According to at least one of the embodiments described above, the effectiveness of the relocation operation can be improved, thereby shortening the operating time of the handling device and improving the throughput of the system.

[0190] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0191] 1...Transport system, 10...Handling device (handling system), 10A...First handling device, 10B...Second handling device, 11...Sensor, 11A...First sensor, 11B...Second sensor, 11C...Third sensor, 11D...Fourth sensor, 11E...Fifth sensor, 12...Control device (control unit), 100... movable Arm, 101...arm member, 102...rotating section, 200...holding section, 200A...first holding section, 200B...second holding section, 202...clamping hand, 203...suction device, 205...suction section, 300...input section, 301...recognition processing section, 302...memory section, 303...operation control section, 304...score calculation section, 305...threshold generation section, 306...judgment section, 307...holding strategy determination section, 308...holding inhibition factor estimation section, 309...arrangement change strategy determination section.

Claims

1. A handling system capable of handling a plurality of objects, A movable arm; a holding unit attached to the movable arm and capable of holding the object by selecting one or more holding methods from a plurality of holding methods; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The control unit determining whether to reposition the object before holding the object based on the information acquired from the sensor; If it is determined that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; the control unit determines an object to be subjected to the rearrangement operation and a holding method of the holding unit to be used for the rearrangement operation based on a result of the evaluation. Handling system.

2. the control unit calculates a score of the rearrangement operation for each of the objects in the evaluation, and selects an object to be rearranged based on the score. Handling system according to claim 1 .

3. the score of the rearrangement operation calculated for each object is calculated based on at least one of the ease of rearranging the target object, the difficulty of holding the target object, and the number of objects expected to be rearranged in association with the rearrangement operation of the target object; Handling system according to claim 2 .

4. the control unit estimates a holding impediment factor that inhibits holding of the object based on the information acquired from the sensor, and determines a placement change operation based on the estimated holding impediment factor and a result of the evaluation.

4. A handling system according to claim 1.

5. the control unit determines a degree of crowding of the objects based on at least one of a layout and a shape of the objects, and determines a layout change operation to reduce the degree of crowding of the objects. Handling system according to any one of claims 1 to 4.

6. A handling system capable of handling multiple objects, A movable arm; a holder attached to the movable arm and capable of holding the object; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The control unit determining whether to reposition the object before holding the object based on the information acquired from the sensor; If it is determined that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; the holding unit is configured to hold the object by at least one of clamping and suction, the control unit determines a placement change operation to increase the distance between the objects when the holding unit holds the objects by pinching, and determines a placement change operation to increase the area available for adsorption of the objects when the holding unit holds the objects by suction. Handling system.

7. the holding unit is capable of holding an object by selecting one or more holding methods from a plurality of holding methods, the holding method currently selected by the holding unit is used as the holding method of the holding unit used in the placement change operation; Handling system according to any one of claims 1 to 6.

8. the control unit calculates a score of ease of holding for each of the objects based on the information acquired from the sensor, and determines whether to change the placement of the objects based on the score of ease of holding.

8. A handling system according to any one of claims 1 to 7.

9. the control unit determines to change the arrangement of the object when there is no object whose holdability score is equal to or greater than a predetermined threshold.

9. A handling system according to claim 8.

10. A handling system capable of handling a plurality of objects, comprising: A movable arm; a holder attached to the movable arm and capable of holding the object; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The control unit determining whether to reposition the object before holding the object based on the information acquired from the sensor; If it is determined that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; the control unit selects one or more types of rearrangement operations from a plurality of types of rearrangement operations based on at least one of the information on the object and the information acquired from the sensor, and evaluates the effectiveness of the rearrangement operation of the selected type for each of the objects. Handling system.

11. A handling system capable of handling a plurality of objects, comprising: A movable arm; a holder attached to the movable arm and capable of holding the object; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The control unit determining whether to reposition the object before holding the object based on the information acquired from the sensor; If it is determined that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; the control unit selects one or more types of rearrangement operations from a plurality of types of rearrangement operations based on at least one of information about the object and information acquired from the sensor, and evaluates effectiveness of the selected type of rearrangement operation for each of the objects; the control unit selects one or more types of rearrangement operations from among a plurality of types of rearrangement operations based on one or more pieces of information selected from the group consisting of flexibility of the object, a surface area of a portion of the object that is expected to be exposed after the rearrangement operation, an aspect ratio of the object, a difference in height between the object and its surroundings, and a size of a space in a peripheral area of the object; Handling system.

12. the control unit calculates a position at which the determined rearrangement operation is to be performed and an attitude of the movable arm for performing the determined rearrangement operation. Handling system according to any one of claims 1 to 11.

13. the control unit calculates a score for determining a priority of a holding operation for each of the objects based on the information acquired from the sensor, and when it is determined that the holding operation by the holding unit has failed, corrects the score for determining the priority based on information about the object for which the holding operation has failed. Handling system according to any one of claims 1 to 12.

14. A conveying system capable of conveying a plurality of objects, A movable arm; a holding unit attached to the movable arm and capable of holding the object by selecting one or more holding methods from a plurality of holding methods; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The control unit determining whether to reposition the object before holding the object based on the information acquired from the sensor; If it is determined that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; If it is determined that the position of the object is not to be changed, a position for holding the object selected as the object to be held and an attitude of the movable arm are calculated, and the object is moved from a first position to a second position; the control unit determines an object to be subjected to the rearrangement operation and a holding method of the holding unit to be used for the rearrangement operation based on a result of the evaluation. Conveying system.

15. A control device for controlling a handling system capable of handling a plurality of objects, The handling system comprises: A movable arm; a holding unit attached to the movable arm and capable of holding the object by selecting one or more holding methods from a plurality of holding methods; a sensor capable of detecting the object; Equipped with The control device determining whether to reposition the object before holding the object based on the information acquired from the sensor; If it is determined that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; the control device determines an object to be rearranged and a holding method of the holding unit to be used for the rearrangement operation based on a result of the evaluation. Control device.

16. A program for controlling a handling system capable of handling a plurality of objects, The handling system comprises: A movable arm; a holding unit attached to the movable arm and capable of holding the object by selecting one or more holding methods from a plurality of holding methods; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The program causes the control unit to determining whether to reposition the object before holding the object based on the information obtained from the sensor; When the control unit determines that the object should be rearranged, evaluating the effectiveness of the rearrangement operation for each of the objects, and determining the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding; Execute the program causes the control unit to execute a step of determining, based on a result of the evaluation, an object to be subjected to a placement change operation and a holding method of the holding unit to be used for the placement change operation. program.

17. A handling method using a handling system capable of handling a plurality of objects, comprising: The handling system comprises: A movable arm; a holding unit attached to the movable arm and capable of holding the object by selecting one or more holding methods from a plurality of holding methods; a sensor capable of detecting the object; a control unit that controls the movable arm and the holding unit; Equipped with The control unit determining whether to reposition the object before holding the object based on the information acquired from the sensor; When the control unit determines that the object should be rearranged, the control unit evaluates the effectiveness of the rearrangement operation for each of the objects, and determines the rearrangement operation based on the results of the evaluation so that the rearrangement operation improves the ease of holding the object; the control unit determines an object to be subjected to the rearrangement operation and a holding method of the holding unit to be used for the rearrangement operation based on a result of the evaluation. Handling method.

Images