Transport system, conveying system, control device, storage medium, and transport method

Abstract

The present invention provides a transport system, a conveying system, a control device, a storage medium, and a transport method that can improve the throughput of the system. The transport system of an embodiment is a transport system capable of transporting multiple objects, and comprises a movable arm, a holding unit, a sensor, and a control unit. The holding unit is mounted on the movable arm and is capable of holding the object. The sensor is capable of detecting the object. The control unit controls the movable arm and the holding unit. Based on information obtained from the sensor, the control unit determines whether to change the configuration of the object before holding the object. If it is determined that the configuration of the object is to be changed, the control unit evaluates the effectiveness of the configuration change operation for each object and determines the configuration change operation based on the evaluation result.

Description

[0001] This application is based on and claims the benefit of priority from Japanese Patent Application No. 2021-094490 (filing date: June 4, 2021), the entire contents of which are incorporated herein by reference. Technical Field

[0002] Embodiments of the present invention relate to a transport system, a conveying system, a control device, a non-volatile computer-readable storage medium storing a program, and a transport method. Background Art

[0003] Conventionally, there are known conveying devices that use end effectors to hold objects. The automation of transfer operations in logistics sites requires the ability to hold objects of various shapes, sizes, and weights. Using conveying devices to hold these objects requires extensive computation to determine holding strategies, including the holding position, holding method, and robot arm posture. Complex loading conditions increase the computational time required to determine the holding strategy.

[0004] Furthermore, when selectively using multiple holding tools, if the number of holding method changes is excessive, the overall operation time may be extended by the amount of tool replacement required. Furthermore, in situations where the probability of holding failure is high, such as when holding objects are densely packed, the overall operation time may be extended by the amount of holding failures. Consequently, the system throughput may be reduced. Summary of the Invention

[0005] Problems to be solved by the invention

[0006] The problem to be solved by the present invention is to provide a transport system, a conveying system, a control device, a program, and a transport method that improve the throughput of the system.

[0007] A transport system according to an embodiment is capable of transporting multiple objects and includes a movable arm, a holding unit, a sensor, and a control unit. The holding unit is mounted on the movable arm and is capable of holding an object. The sensor is capable of detecting the object. The control unit controls the movable arm and the holding unit. Based on information obtained from the sensor, the control unit determines whether to reposition the object before holding it. If a repositioning operation is determined to be necessary, the control unit evaluates the effectiveness of the repositioning operation for each object and, based on the evaluation results, determines whether to perform the repositioning operation.

[0008] According to the conveying system having the above-described structure, the throughput of the system can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1It is a perspective view schematically showing a conveying system including the transport system according to the first embodiment.

[0010] Figure 2 This is a block diagram showing the system configuration of a transport system including the conveying system according to the first embodiment.

[0011] Figure 3 This is a control flowchart of the control device according to the first embodiment.

[0012] Figure 4 3 is a diagram showing a temporarily blocked area in the image data of the object when the object O is held by “gripping”.

[0013] Figure 5 is a diagram showing an occluded area in image data of an object.

[0014] Figure 6 It is a graph representing the depth image of an object.

[0015] Figure 7 A diagram that illustrates the three-dimensional position and orientation of an object.

[0016] Figure 8 This is a diagram showing a masked area to which information related to three-dimensional position and orientation is added.

[0017] Figure 9 This is a control flowchart of the maintenance strategy planning process performed by the control device according to the first embodiment.

[0018] Figure 10 This is a schematic diagram showing an example of a configuration change operation.

[0019] Figure 11 This is a schematic diagram showing an example of a configuration change operation.

[0020] Figure 12 This is a schematic diagram showing an example of a configuration change operation.

[0021] Figure 13 This is a schematic diagram showing an example of a configuration change operation.

[0022] Figure 14 This is a control flow chart of the control device according to the second embodiment.

[0023] Figure 15 This is a control flowchart of a holding strategy planning step of a control device according to a third embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, a transport system, a conveying system, a control device, a storage medium, and a transport method according to an embodiment will be described with reference to the drawings.

[0025] In addition, in the following description, the same reference numerals are used for structures having the same or similar functions. In addition, repeated descriptions of these structures are sometimes omitted. In addition, "based on XX" recorded in this application means "at least based on XX", and also includes situations based on other elements in addition to XX. In addition, "based on XX" is not limited to the situation of directly using XX, but also includes situations based on calculations and processing of XX. "XX" is an arbitrary element (such as arbitrary information).

[0026] (First embodiment)

[0027] Reference Figures 1 to 13 , an implementation method is described. Figure 1 It is a perspective view schematically showing a transport system 1 including a transport device 10 (an example of a “transport system”) according to the present embodiment.

[0028] Conveyor system 1 is, for example, a transport system (sorting system) for logistics. Conveyor system 1 moves objects (held objects, transported objects) O located at a source V1 toward a destination V2. For example, conveyor system 1 removes a specified number of various objects O from the source V1 and loads them onto the destination V2.

[0029] The transport source V1 is, for example, a conveyor, a pallet, or a container such as a tote or collapsible container. "Container" broadly refers to a member (e.g., a box-shaped member) capable of storing an object O. However, the transport source V1 is not limited to the examples above. In the following description, the "transport source V1" may be referred to as the "removal source container V1."

[0030] A variety of objects O of varying sizes and weights are randomly placed on the mobile source V1. For example, the objects O to be held may have a concave-convex shape on at least a portion of their surface. In this embodiment, the outer shapes of the objects O vary, ranging from as small as a 5 cm square to as large as a 30 cm square. Furthermore, the objects O range from as light as tens of grams to as heavy as several kilograms. However, the size and weight of the objects O are not limited to the above examples.

[0031] The destination V2 is, for example, a container such as a tote or a collapsible container. However, the destination V2 is not limited to the above examples. In the following description, the "destination V2" may be referred to as the "destination container V2," and the "source V1" and "destination V2" may be collectively referred to as the "container." Furthermore, the transport system 1 may also transport the object O to a destination V2 other than a container.

[0032] The conveyor system 1 is not limited to a transport system for logistics. The conveyor system 1 can also be widely applied to industrial robot systems and other systems. The "conveyor system," "transportation system," and "transportation device" described in this application are not limited to systems and devices whose primary purpose is to transport objects. They also include systems and devices that transport (move) objects as part of product assembly or other purposes.

[0033] like Figure 1 As shown, the transport system 1 includes a conveying device 10 , a sensor 11 , and a control device 12 (an example of a “control unit”). The control device 12 can be incorporated into the conveying device 10 .

[0034] The transport device 10 is, for example, a robotic device. The transport device 10 holds an object O located in a removal source container V1 and moves the held object O to a destination container V2 (storage area). The transport device 10 can communicate with the control device 12 via wired or wireless communication. In this embodiment, the transport device 10 includes a first transport device 10A and a second transport device 10B.

[0035] The first transport device 10A includes, for example, a movable arm 100 and a first holding portion 200A provided at the front end of the movable arm 100 .

[0036] The movable arm 100 is a moving mechanism that moves the first holding portion 200A to a desired position. For example, the movable arm 100 is a six-axis vertical multi-jointed robotic arm. The movable arm 100 can assume various positions and postures. Similar to a human arm or hand, the movable arm 100 can also adopt a wider variety of postures for holding an object. The movable arm 100 includes, for example, multiple arm components 101 and multiple rotating parts 102 that rotatably connect the multiple arm components 101.

[0037] The movable arm 100 may also be a three-axis orthogonal robot arm. Alternatively, the movable arm 100 may utilize other structures to move the first holding portion 200A to a desired position. For example, the movable arm 100 may be an aerial vehicle (e.g., a drone) that uses rotating wings to lift and move the first holding portion 200A.

[0038] The first holding unit 200A is a holding mechanism (end effector) that holds the object O located in the removal source container V1 . The first holding unit 200A includes a gripping robot 202 .

[0039] The gripper robot 202 is a gripper-type robot that grips and holds the object O with two fingers and is provided at the front end of the movable arm 100. The structure of the gripper robot 202 is not limited thereto and may be a gripper-type robot that grips and holds the object O with three fingers, for example.

[0040] Alternatively, the first holding unit 200A may be a hybrid manipulator that includes a suction device and a suction unit connected to the suction device, and holds the object O by gripping and / or suction. In this case, the suction unit may be provided at the tip of the fingers of the gripping manipulator 202. Alternatively, a plurality of suction units may be provided at the tip of the fingers of the gripping manipulator 202.

[0041] The second transport apparatus 10B includes, for example, an arm (second arm) 100 and a second holding portion 200B provided at the distal end of the movable arm 100. The movable arm 100 of the second transport apparatus 10B has the same structure as the movable arm 100 of the first transport apparatus 10A.

[0042] The second holding unit 200B is a holding mechanism (end effector) that holds the object O located in the extraction source container V1. For example, the second holding unit 200B includes a suction device 203 and a suction unit 205 connected to the suction device 203. The second holding unit 200B is a suction-type robot that holds the object O by suction.

[0043] The second holding portion 200B may also be a mechanism that holds the object O using other holding methods. For example, the second holding portion 200B may be a holding portion that can use magnetic force to hold the object O. For example, the second holding portion 200B may be a holding portion that can use a jamming phenomenon to hold the object O (e.g., a jamming gripper) and is composed of a soft film filled with powder and a vacuum pump that extracts air from the soft film.

[0044] The suction device 203 is, for example, a vacuum pump and is connected to each of the plurality of adsorption units 205 via a hose or the like. By driving the suction device 203 , the pressure in each adsorption unit 205 becomes lower than atmospheric pressure, and the adsorption unit 205 adsorbs and holds the object O.

[0045] The suction unit 205 is provided at the front end of the second holding unit 200B. For example, multiple suction units 205 may be provided at the front end of the second holding unit 200B. The suction units 205 have an outer shape smaller than the smallest object O located in the removal source container V1. The second transport device 10B suction-holds the object O using only one or more suction units 205 selected from the plurality of suction units 205.

[0046] In the following description, the "first holding part 200A" and the "second holding part 200B" are collectively referred to as the "holding part 200". That is, the "holding part 200" includes the "first holding part 200A" and the "second holding part 200B". In addition, the first holding part 200A is described as a clamping-type manipulator and the second holding part 200B is described as an adsorption-type manipulator, but the structure of the holding part 200 is not limited to the structure having the first holding part 200A and the second holding part 200B as a suction-type manipulator as described above. In this embodiment, the case where both the first holding part 200A and the second holding part 200B are clamping-type manipulators or adsorption-type manipulators is also included. In this case, the holding part 200 may also be a structure having a plurality of clamping-type manipulators having different characteristics in at least one of the structure, construction, shape, size, and configuration. Specifically, for example, the first holding portion 200A and the second holding portion 200B may be two or more clamping-type manipulators having different claw lengths or opening widths. In addition, the holding portion 200 may be, for example, a structure of multiple adsorption-type manipulators having different characteristics in at least one of the structure, construction, shape, size, and configuration. Specifically, for example, the first holding portion 200A and the second holding portion 200B may be two or more adsorption-type manipulators having different configurations of adsorption pads, diameters of adsorption pads, and structures of bellows. Even in this case, the present embodiment can be implemented in the same manner and achieve the same effect.

[0047] The sensor 11 is controlled by the control device 12 and can thereby detect the status of multiple objects O and / or the holding portion 200. The sensor 11 includes a first sensor 11A, a second sensor 11B, a third sensor 11C, a fourth sensor 11D, and a fifth sensor 11E. The first sensor 11A, the second sensor 11B, the third sensor 11C, the fourth sensor 11D, and the fifth sensor 11E are connected to the control device 12 via a wired or wireless connection. The first sensor 11A to the fifth sensor 11E do not necessarily need to be independent sensors; a specific sensor can independently perform the functions of multiple sensors among the first sensor 11A to the fifth sensor 11E.

[0048] The first sensor 11A is a camera or various sensors that are arranged near the moving source V1 (for example, directly above or diagonally above the moving source V1). The first sensor 11A obtains, for example, information related to the object O located at the moving source V1 and information related to the moving source V1. The information obtained by the first sensor 11A is, for example, "image data", "distance image data", "shape data", etc. "Distance image data" refers to image data with distance information in one or more directions (for example, depth information from an arbitrary reference plane set above the moving source V1). "Shape data" is information indicating the outer shape of the object O, etc. The information detected by the first sensor 11A is output to the control device 12. In addition, the first sensor 11A can also be provided as a part of the conveying device 10.

[0049] The second sensor 11B is a camera or other type of sensor located near the destination container V2 (e.g., directly above or diagonally above the destination container V2). The second sensor 11B detects, for example, information related to the shape of the destination container V2 (including the shape of the inner wall and partitions) and information related to the object O previously placed in the destination container V2. Examples of information acquired by the second sensor 11B include "image data," "distance image data," and "shape data." The information detected by the second sensor 11B is output to the control device 12. Alternatively, the second sensor 11B may be provided as part of the transport device 10.

[0050] The third sensor 11C is a variety of sensors provided near the first holding portion 200A or the first holding portion 200A. The third sensor 11C acquires information related to the physical state of the first holding portion 200A, such as deformation of the first holding portion 200A, pressure applied to the first holding portion 200A, and the surface condition of the first holding portion 200A. The third sensor 11C may include, for example, one or more physical sensors such as a deformation sensor, a pressure sensor, and a proximity sensor. The third sensor 11C may also acquire physical information about the object O. The information detected by the third sensor 11C is output to the control device 12. Alternatively, the third sensor 11C may be provided as part of the transport device 10.

[0051] The fourth sensor 11D is one of various sensors provided near the second holding portion 200B or the second holding portion 200B. The fourth sensor 11D acquires information related to the physical state of the second holding portion 200B, such as deformation of the second holding portion 200B, pressure applied to the second holding portion 200B, and the surface condition of the second holding portion 200B. The fourth sensor 11D may include, for example, one or more physical sensors such as a deformation sensor, a pressure sensor, and a proximity sensor. The fourth sensor 11D may also acquire physical information about the object O. The information detected by the fourth sensor 11D is transmitted to the control device 12. Alternatively, the fourth sensor 11D may be provided as part of the transport device 10.

[0052] The fifth sensor 11E acquires information related to the usage status of the holding section 200. For example, the fifth sensor 11E detects the holding section 200 currently in use or selected for use among the first holding section 200A and the second holding section 200B (hereinafter, the holding sections currently in use or selected for use are sometimes collectively referred to as "currently selected holding sections" or simply "selected holding sections"). The information detected by the fifth sensor 11E is output to the control device 12. In addition, the fifth sensor 11E may also be provided as a part of the conveying device 10. In addition, the currently selected holding section 200 may be determined not based on the fifth sensor 11E but based on other information such as the control history of the conveying device 10, information acquired by the third sensor 11C or the fourth sensor 11D, or the like. In this case, the fifth sensor 11E can be omitted. Moreover, in the case where the holding portion 200 is a manipulator capable of performing clamping, adsorption, and a hybrid of clamping and adsorption, the control device 12 determines whether the holding portion 200 is in a state suitable for clamping the object O, a state suitable for adsorbing the object O, or a state suitable for both based on information from the 5th sensor 11E or other information.

[0053] The control device 12 manages and controls the entire conveying system 1. For example, the control device 12 acquires information detected by the first sensor 11A to the fifth sensor 11E and controls the conveying device 10 based on the acquired information. The control device 12 is, for example, a programmable device (computer) having a processor, memory, storage device, etc.

[0054] Figure 2 1 is a block diagram showing the system configuration of the transportation system 1 .

[0055] The control device 12 is directly or indirectly connected to the first sensor 11A to the fifth sensor 11E, the movable arm 100, the first holding unit 200A, and the second holding unit 200B via wired or wireless communication. The control device 12 includes an input unit 300, a recognition processing unit 301, a storage unit 302, an action control unit 303, a score calculation unit 304, a threshold generation unit 305, a determination unit 306, a holding strategy determination unit 307, a holding obstacle estimation unit 308, and a configuration change strategy determination unit 309.

[0056] The input unit 300 receives an instruction list regarding the object O to be held from an operator or the system, information acquired by the first to fifth sensors 11A to 11E, and the like.

[0057] The recognition processing unit 301 processes information acquired by the first to fifth sensors 11A to 11E. For example, the recognition processing unit 301 determines the position, posture, shape, characteristics, etc. of the object O at the movement source V1 based on the image data acquired by the first sensor 11A.

[0058] The storage unit 302 stores control programs for the control device 12, shape data related to the objects O to be sorted, instruction lists received from operators or the system, various scores generated by the score calculation unit 304, thresholds generated by the threshold generation unit 305, determination results by the determination unit 306, the holding strategy determined by the holding strategy determination unit 307, the holding obstacle estimated by the holding obstacle estimation unit 308, the relocation strategy determined by the relocation strategy determination unit 309, and the control and operation history of the transport device 10. The shape data stored in the storage unit 302 is defined by the local coordinate system of the object O.

[0059] The motion control unit 303 controls the motion of the movable arm 100, the first holding unit 200A, the second holding unit 200B, and the like of the transport device 10. For example, the motion control unit 303 specifically calculates the position at which a specific holding unit holds a specific object O and the posture of the movable arm 100 during holding. Based on the calculated position at which the object O is held and the posture of the movable arm 100 during holding, the motion control unit 303 instructs the transport device 10 to cause the first holding unit 200A or the second holding unit 200B to perform a holding operation for the specific object O. Furthermore, the motion control unit 303 specifically calculates the position at which the specific holding unit changes the configuration of the object O and the posture of the movable arm 100 during the configuration change. The motion control unit 303 instructs the transport device 10 to cause the first holding unit 200A or the second holding unit 200B to perform a specific object O relocation operation based on the calculated position for relocation of the object O and the posture of the movable arm 100 during the relocation.

[0060] In this specification, a "configuration change operation" refers to a physical operation that uses physical force to change the spatial configuration of a specific object. The content of the configuration change operation is determined by the object, the specific force applied to the object, the means for applying the force to the object, and the specific method for changing the spatial configuration of the object.

[0061] The score calculation unit 304 generates scores for determining the object O to be held, the priority of the holding method, the object O to be changed, the effectiveness of the change operation, etc. For example, the score calculation unit 304 calculates the first score, the second score, and the third score described below.

[0062] The threshold generation unit 305 generates a threshold for determining whether a holding method needs to be switched, whether a configuration needs to be changed, etc. For example, the threshold generation unit 305 generates a first threshold for a first score and a second threshold for a second score, which will be described later.

[0063] The determination unit 306 determines whether the holding method needs to be switched, whether the arrangement of the object O needs to be changed, etc., based on the score generated by the score calculation unit 304 and the threshold generated by the threshold generation unit 305 .

[0064] The retention strategy determination unit 307 determines a retention strategy including a retention object and retention method with high priority, the order of retaining the retention objects, the order of retaining actions and switching actions of the retention methods, etc. based on the score generated by the score calculation unit 304, the threshold generated by the threshold generation unit 305, the result of the judgment of the judgment unit 306, etc.

[0065] The holding-impeding factor estimation unit 308 estimates the presence of various holding-impeding factors that hinder the holding unit 200 from holding the object O based on information acquired by the first sensor 11A to the fifth sensor 11E. For example, the holding-impeding factor estimation unit 308 estimates the presence of holding-impeding factors based on image data acquired by the first sensor 11A and the recognition result output by the recognition processing unit 301.

[0066] The configuration change strategy determination unit 309 determines the configuration change strategy based on the score generated by the score calculation unit 304, the threshold generated by the threshold generation unit 305, the judgment result of the judgment unit 306, etc., so as to change the configuration of object O so as to make it easier to maintain object O when it is difficult to maintain object O.

[0067] All or part of the various functions of the control device 12 are implemented, for example, by one or more processors, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), executing programs stored in a program memory. However, all or part of these functions may also be implemented by hardware (e.g., circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a PLD (Programmable Logic Device). Furthermore, all or part of the above functions may also be implemented by a combination of software and hardware. The storage unit 302 is implemented by flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), or RAM (Random Access Memory).

[0068] Next, the operation of the conveying system 1 will be described. Figure 3 The control flow chart of the control device 12 shown in FIG.

[0069] When the control device 12 is activated, the control device 12 initializes the conveying device 10 and the sensor 11 and then starts controlling the conveying device 10 (step S0 ). Next, the control device 12 executes step S1 .

[0070] In step S1 , the input unit 300 of the control device 12 receives an instruction list of objects O to be sorted from an operator or the system (an instruction list receiving step).

[0071] Next, the control device 12 executes step S2. In step S2, the input unit 300 of the control device 12 receives image data, distance image data, shape data, and other information related to the object O in the removal source container V1 from the first sensor 11A. Based on the data received by the input unit 300, the recognition processing unit 301 of the control device 12 determines whether an object O listed in the instruction list is present in the removal source container V1. Furthermore, based on the data received by the input unit 300, the recognition processing unit 301 acquires information related to the shape, position, and posture of the object O to be sorted (information acquisition step).

[0072] Figure 4 1 is a diagram showing a temporary mask region R1 in image data of an object O when the object O is held by “gripping”.

[0073] The recognition processing unit 301 of the control device 12 uses a known image segmentation method to set the rectangular area circumscribing the object O to be sorted (circumscribed rectangular area) as the "temporary mask area R1" based on the image data. Image segmentation can be a technique using machine learning.

[0074] Figure 5 3 is a diagram showing an obscured area R2 in image data of an object O.

[0075] The recognition processing unit 301 sets the area expanded in the longitudinal and transverse directions of the temporary shielding area R1 as "shielded area R2." Shielding area R2 is expanded in the longitudinal and transverse directions by a margin M of the circumscribed rectangular area. For example, margin M is 100 mm. Using this expanded shielding area R2, the recognition processing unit 301 can determine whether there is space around the object O that the gripping robot 202 can enter, when the gripping method of the holding unit 200 is gripping.

[0076] Figure 6 is a diagram showing a depth image D of an object O.

[0077] The recognition processing unit 301 uses the range image data to generate a depth image D that visualizes the depth information of the object O in the masked area R2. The depth image D has values ​​representing the height relative to the origin of the world coordinate system (X-axis, Y-axis, and Z-axis). The scale of the depth image D can be changed based on the storage method of the storage unit 200. For example, the scale of the depth image D can be set to 1 mm per pixel.

[0078] Figure 7 is a diagram illustrating the three-dimensional position and posture of object O. Figure 7In the diagram, objects O with different shapes are recorded as "O1" and "O2".

[0079] The recognition processing unit 301 calculates the three-dimensional position and posture of the object O1 based on the acquired image data of the object O1. The recognition processing unit 301 converts the shape data of the object O1 in the local coordinate system recorded in the storage unit 302 into the world coordinate system (X axis, Y axis, Z axis) using the transformation matrix. Figure 7 As shown, the Z-axis direction of the world coordinate system is the depth direction measured from an arbitrary reference plane set above the extraction source container V1. The recognition processing unit 301 calculates the three-dimensional position and posture of the object O1 by comparing the acquired image data of the object O1 with the shape data converted into the world coordinate system.

[0080] Figure 8 This diagram shows the masked area R2, to which information related to the three-dimensional position and posture is added. The recognition processing unit 301 sets the three-dimensional position pose, which serves as the reference for the object O1, as the center CO of the circumscribed rectangular area (temporary masked area R1) in the masked image R2. Alternatively, the recognition processing unit 301 may calculate the centroid FO of the object O1 outside the center CO of the circumscribed rectangular area, taking into account the gripping ease of the gripping robot 202, and utilize this.

[0081] On the other hand, when holding an object O by "adsorption," a temporary shielding region R1 is segmented according to each shielding plane on the surface of object O that can serve as an adsorption target. The recognition processing unit 301 generates a shielding image R2 and a depth image D for each shielding plane, defining the direction perpendicular to the shielding plane as the normal direction. The recognition processing unit 301 extracts a plane region from the point cloud of the three-dimensional sensor, for example, and determines the position and posture by performing principal component analysis, such as by setting the short axis direction to x, the long axis direction to y, and the plane normal direction to z. In the case of "clamping," only one holding region may be defined for one object O, but in the case of "adsorption," multiple holding regions may exist for one object O. Therefore, the holding target of "adsorption" can be a single region of object O. Furthermore, for example, in the case of multiple holding regions for "clamping," the holding target of "clamping" can also be a single region of object O.

[0082] When the object O is held by “adsorption”, the recognition processing unit 301 can perform convolution processing for calculating how many of the plurality of adsorption units 205 to use, at which angle the adsorption unit 205 is brought into contact with the object O, and the like.

[0083] The recognition processing unit 301 may use a database that records the three-dimensional position and posture of the object O when the object O was previously successfully sorted. The recognition processing unit 301 may also output a recommended holding method and holding position for the object O using the database.

[0084] The input unit 300 receives information related to the physical state of the first holding unit 200A from the third sensor 11C as needed, and receives information related to the physical state of the second holding unit 200B from the fourth sensor 11D. In addition, the input unit 300 receives information related to the usage status of the holding unit 200, such as the currently selected holding unit, from the fifth sensor 11E as needed. The recognition processing unit 301 determines which of the first holding unit 200A and the second holding unit 200B is the currently selected holding unit 200 based on the information from the fifth sensor 11E and / or other information. In addition, the information related to the physical state of the first holding unit 200A and the second holding unit 200B and the information related to the currently selected holding unit 200 may be appropriately acquired at other timings other than step S2.

[0085] Furthermore, the input unit 300 receives physical information of the object O to be held from a database. For example, the input unit 300 can receive information related to the external shape, weight, surface material, friction characteristics, etc. of the object O from the database. The database can be defined for each individual object O, or it can specify basic information primitives (e.g., a cuboid, cylinder, sphere, pyramid, etc.) and use information that approximates their shape.

[0086] Next, the control device 12 executes step S3. In step S3, the control device 12 calculates a score for each object O or region of object O, and for each retention method, and determines the next object O to be retained and the retention method (retention strategy planning step). If retention is predicted to be difficult, the control device 12 determines the details of the relocation operation for the object O. Figure 9 4 is a control flow chart of the holding strategy planning process performed by the control device 12, and shows the details of step S3.

[0087] In step S301, the score calculation unit 304 of the control device 12 calculates the ease of holding the object O by the holding unit 200 as a "first score" based on the information obtained in step S2. The score calculation unit 304 calculates the first score for (1) each object O or a region of the object O (hereinafter also referred to as "each object O") and (2) each holding method. For example, the holding method is "gripping" performed by the first transport device 10A or "suction" performed by the second transport device 10B. When the holding method is "gripping", the first score is calculated for each object O, for example. When the holding method is "suction", the first score is calculated for each region of the object O, for example.

[0088] The first score S calculated in step S301 H (I) is calculated, for example, using the evaluation function shown in Formula 1.

[0089] [Formula 1]

[0090]

[0091] In Formula 1, H is the holding method (clamping by the first transport device 10A, suction by the second transport device 10B, etc.) that is the subject of the first score evaluation. I is the shielded area R2 representing the object O or the area of ​​the object O that is the subject of the first score evaluation. Hi is the evaluation item of the evaluation function, w Hi is the weight of the evaluation function. That is, the above-mentioned evaluation function is represented by a linear combination of the evaluation items. However, the evaluation function is not limited to the above example; an average value or a nonlinear function can be used for the evaluation items; any evaluation function can be used. Furthermore, the evaluation items are not limited to those described in Formula 1; for example, they can also be quantities that depend on the physical properties of the object O or the gripping robot 202.

[0092] When the holding method is "clamp" (assuming H = p), the first score S p (I) is calculated, for example, using the evaluation function shown in Formula 2.

[0093] [Formula 2]

[0094]

[0095] In equation 2, w p1 +w p2 +w p3 +w p4 +w p5 +w p6 =1,w pi ≥0 (i=1, 2, 3, 4, 5, 6). fp1 is the position of object O (e.g., the center position of object O and the height of object O). p2 is the concavity and convexity of object O. f p3 is the flatness of object O. f p4 It is the difference in depth between object O and its surroundings. p5 Is the result of judging whether the object O is thin. p6 It is an evaluation item used when using machine learning, combining the above f p1 ~f p5 Therefore, for example, when using f p6 In the case of p1 ~w p5 All can be set to zero. Calculate the first score S of "clamping" for each occlusion area I representing each object O. p (I).

[0096] When the retention method is "adsorption" (assuming H = s), the first score S s (I) is calculated, for example, using the evaluation function shown in Formula 3.

[0097] [Formula 3]

[0098]

[0099] In equation 3, w s1 +w s2 +w s3 +w s4 +w s5 +w s6 =1,w si ≥0 (i=1, 2, 3, 4, 5, 6). f s1 is the position of the region of object O (for example, the center position of the region of object O and the height of object O). s2 is the concavity and convexity of the region of object O. f s3 is the area of ​​the region of object O. f s4 It is the approach angle of the region relative to the object O (for example, the angle between the normal direction of the region of the object O and the vertical direction). s5 Is the result of judging whether the object O is thin. s6 It is an evaluation item used when using machine learning, combining the above f s1 ~f s5 Therefore, for example, when using f s6 In the case of s1 ~w s5 All of them can be zero. Calculate the first score S of "adsorption" for various shielding areas I representing the area of ​​the object O.s (I).

[0100] In this way, the score calculation unit 304 calculates the first score S for each object O or region of the object O with respect to each holding method (here, clamping by the first transport device 10A and adsorption by the second transport device 10B). H (I).

[0101] Next, the control device 12 executes step S302. In step S302, the determination unit 306 of the control device 12 determines whether each first score S calculated in step S301 is H (I) Whether there is a score greater than a predetermined first threshold value Th1. H If at least one of (I) is greater than or equal to the first threshold value Th1 (step S302: YES), the control device 12 proceeds to step S303 in order to plan a holding strategy for the object O. On the other hand, if it is determined that all the first scores S calculated for each holding method and each object O or region of the object O are H (I) When the value is lower than the first threshold value Th1 (step S302 : No), the control device 12 proceeds to step S310 in order to plan an arrangement change strategy that facilitates the holding of the object O.

[0102] In step S302, it is determined that there is a first score S greater than the first threshold Th1. H In the case of (I) (step S302: Yes), the control device 12 executes step S303. In step S303, the score calculation unit 304 of the control device 12 calculates a "second score" based on the currently selected holding method based on the information acquired in step S2 and the first score calculated in step S301. The score calculation unit 304 calculates the second score for (1) each object O or region of object O and (2) each holding method.

[0103] The second score T calculated in step S303 H，H0 (I) is calculated, for example, using the evaluation function shown in Formula 4.

[0104] [Formula 4]

[0105]

[0106] In Formula 4, H is the holding method to be evaluated for the second score. H0 is the currently selected holding method. I is the masked area R2 representing the object O or the area of ​​the object O to be evaluated for the second score. In other words, the evaluation function of Formula 4 is the first score S of the currently selected holding method. H0(I) The first score S of the maintenance method as the evaluation target of the second score H However, the evaluation function of the second score is not limited to the above example, and any function can be used, for example, it can be the first score S that depends on the currently selected holding method H0. H0 (I) and the first score S of the holding method H as the evaluation target of the second score H (I) is an arbitrary function.

[0107] 1st Score S H (I) is an index of the “ease of maintaining” of the target shielded area I based on the target maintaining method H. Therefore, when the evaluation function of the above-mentioned formula 4 is used, the second score T H，H0 (I) is the ratio of the ease of maintenance based on the currently selected maintenance method H0 to the ease of maintenance based on the object maintenance method H. That is, when the second score of Formula 4 is greater than 1, the currently selected maintenance method H0 is easier to maintain than the object maintenance method H. When the second score of Formula 4 is less than 1, the object maintenance method H is easier to maintain than the currently selected maintenance method H0. For example, when the second score of Formula 4 is 0.5, it can be considered that the ease of maintenance based on the currently selected maintenance method H0 is 0.5 times the ease of maintenance based on the object maintenance method H. That is, it can be considered that the ease of maintenance based on the object maintenance method H is twice the ease of maintenance based on the currently selected maintenance method H0.

[0108] Next, the control device 12 executes step S304. In step S304, the determination unit 306 of the control device 12 determines the value of the second score T H，H0 (I) to determine the necessity of switching the holding method. H，H0 In the case of (I), for example, when the conditional expression of the following formula 5 is satisfied, the judgment unit 306 judges that it is necessary to switch the holding method from the currently selected holding method H0 to the holding method H as the object; when the conditional expression of the following formula 5 is not satisfied, the judgment unit 306 judges that it is not necessary to switch the holding method from the currently selected holding method H0 to the holding method H as the object.

[0109] [Formula 5]

[0110]

[0111] In Formula 5, Th2 is a predetermined second threshold value generated by the threshold value generating unit 305. That is, the determining unit 306 determines the second score T H，H0(I) If the score is lower than the second threshold value Th2, it is determined that it is necessary to switch from the currently selected holding method H0 to the target holding method H. H，H0 (I) When the value is equal to or greater than the second threshold value Th2, it is determined that there is no need to switch from the currently selected holding method H0 to the target holding method H.

[0112] Based only on the first score S H (I) When selecting a holding method, the information of the currently selected holding method H0 will not be reflected in the first score S H (I), therefore, the holding method with the highest success rate is simply selected. Therefore, depending on the situation, it may be determined that the holding method needs to be switched every time the object O is held. In this case, switching the holding method may take time. On the other hand, the second score T H，H0 (I) contains information about the currently selected holding method H0 and is a score calculated based on the currently selected holding method H0. For example, when using the second score T in the above formula 4 H，H0 In the case of (I), even in the second score T H，H0 (I) is less than 1 (that is, when the holding ease based on the target holding method H is greater than the holding ease based on the currently selected holding method H0), the judgment unit 306 sets the second score T H，H0 (I) If the value is not too small (i.e., it is greater than the predetermined second threshold value Th2), it is determined that the holding method does not need to be switched. Thus, when the holding method H0 is relatively easy to use, the control device 12 can control the transport device 10 so that the currently selected holding method H0 is used as much as possible to hold the device 10.

[0113] The determination unit 306 may determine the object O or the region of the object O based on the second score T calculated for each shielded region I. H，H0 The necessity of switching the holding method can be determined based on the magnitude relationship between the average value of (I) and the second threshold value Th2. Alternatively, the necessity of switching the holding method can be determined based on the second score T for each mask area I. H，H0 (I) and the second threshold value Th2 to determine the necessity of switching the holding method. For example, in the above example, the determination unit 306 can determine the necessity of switching the holding method based on the second score T for each shielding area I. H，H0 When at least one of (I) is smaller than the second threshold value Th2, it is determined that the holding method needs to be switched.

[0114] Next, if it is determined that the holding method does not need to be switched (step S305: No), the control device 12 executes step S306. In step S306, the holding strategy determination unit 307 of the control device 12 determines a holding strategy including the order in which the plurality of objects O to be held are held. If the currently selected holding method H0 is "clamping", the holding strategy determination unit 307 can, for example, set the order of holding the objects O to the first score S for "clamping". p (I) Order from largest to smallest. In addition, when the currently selected holding method H0 is "adsorption", the holding strategy determination unit 307 can, for example, set the order of holding the object O to the first score S for "adsorption". s (I) Order from largest to smallest. That is, the holding strategy decision unit 307 can decide the following holding strategy: do not switch the currently selected holding method H0, for example, according to the first score S H0 (I) Pick up the object O in descending order using the currently selected holding method H0.

[0115] Next, the control device 12 executes step S308 . In step S308 , the control device 12 stores the retention strategy including the retention order determined by the retention strategy determination unit 307 in step S306 in the storage unit 302 .

[0116] Next, the control device 12 executes step S309. In step S309, the holding strategy decision unit 307 of the control device 12 decides the next holding action. For example, the holding strategy decision unit 307 determines the first holding action (in the above example, the first holding action of the holding strategy determined in step S306) by the currently selected holding method H0 to hold the first score S H0 (I) The largest object (2) is set as the next holding operation. Then, the control device 12 proceeds to step S4.

[0117] On the other hand, if it is determined that the holding method needs to be switched (step S305: Yes), the control device 12 executes step S307. In step S307, the holding strategy determination unit 307 determines a holding strategy that includes the order in which holding operations and holding method switching operations are performed on the multiple objects O to be held. For example, the holding strategy determination unit 307 can determine the order in which holding operations and holding method switching operations are performed as follows.

[0118] (1) For the first score S based on the currently selected holding method H0 H0 (I) Compared to the first score S based on the holding method H1 after switching H1(I) A large object O or an area of ​​object O, according to the first score S H0 (I) Perform the holding action in order from largest to smallest.

[0119] (2) The holding method is switched from the currently selected holding method H0 to the holding method H1 as the switching target.

[0120] (3) For the first score S based on the currently selected holding method H0 H0 (I) Compared to the first score S based on the holding method H1 after switching H1 (I) Small object O or area of ​​object O, according to the first score S H1 (I) Perform the holding action in order from largest to smallest.

[0121] This makes it possible to create a holding strategy that can pick up all objects O to be held by switching the holding method once, and to minimize the number of times the holding method is switched.

[0122] In addition, the method of determining the holding strategy is not limited to the above example. For example, the holding strategy determination unit 307 may also determine the holding strategy in the following manner: H1，H0 (I) After the object to be held is held by the currently selected holding method H0, the holding method is switched and the second score T is set. H，H0 (I) The retention targets whose values ​​are lower than the second threshold value Th2 are retained using the switched retention method H1. The retention policy determination unit 307 may determine the retention policy using any other method.

[0123] Next, the control device 12 executes step S308 . In step S308 , the control device 12 stores the holding strategy including the order of holding and switching of the holding methods determined by the holding strategy determination unit 307 in step S307 in the storage unit 302 .

[0124] Next, the control device 12 executes step S309. In step S309, the holding strategy determination unit 307 of the control device 12 determines the next holding action. For example, the holding strategy determination unit 307 sets the first holding action in the holding strategy determined in step S307 as the next holding action. For example, in the examples shown in (1) to (3) above, the holding strategy determination unit 307 can set the action of holding the following object O or the area of ​​object O by the currently selected holding method H0 as the next holding action, the object O or the area of ​​object O being based on the first score S of the currently selected holding method H0. H0 (I) Compared to the first score S based on the holding method H1 after switching H1(I) First score S in a large object O or in a region of object O H0 (I) The largest object O or the area of ​​the object O. In addition, if there is no first score S based on the currently selected holding method H0 H0 (I) Compared to the first score S based on the holding method H1 after switching H1 (I) In the case of a large object O or a region of the object O, the holding strategy determination unit 307 can use the switched holding method H1 to hold the first score S based on the switched holding method H1. H1 (I) The action of the largest object O or the area of ​​the object O is set as the next holding action. Then, the control device 12 proceeds to step S4.

[0125] In the steps so far, the control device 12 preferably determines the next holding operation without specifically calculating the position of the held object O, the posture of the movable arm 100, and the like.

[0126] In the above example, the control device 12 selects the first holding action as the next action after determining the overall holding order. However, the holding order can be omitted and only the next action can be determined. For example, the holding strategy determination unit 307 can select the first score S in step S306 without determining the holding order. H (I) The largest action is taken as the next action.

[0127] When the control device 12 determines the holding strategy in step S306 or step S307, the first score S may be calculated by the weight of the object O. H (I) or the second score T H，H0 (I) Correction score after correction.

[0128] In addition, as described above, and are used to calculate the first score S in "clamping" and "adsorption". H The first score S is the same as the evaluation item (I) or the assumed holding object (object O or area of ​​object O, etc.). H The calculation basis for (I) may differ. Therefore, the control device 12 may appropriately normalize the scores so that even scores for different retention methods can be compared. The control device 12 may perform score normalization in any of the steps, such as step S301 for calculating the first score, step S303 for calculating the second score, or steps S306 and S307 for determining the retention strategy.

[0129] The above describes the details of the holding strategy plan in step S3. Here, as an example, the following Tables 1 to 3 show the second score T based on the above formula 4 when the selectable holding method is either "clamping" or "adsorption", the currently selected holding method H0 is "clamping", and the second threshold Th2 is set to 0.5. H，H0 (I) Calculation example. Tables 1 to 3 are tables showing an example of score calculation. Tables 1 to 3 show, for holding objects 1 to 5 representing an object O or an area of ​​an object O, a first score indicating the ease of holding based on the holding method "clamping", a first score indicating the ease of holding based on the holding method "adsorption", a second score calculated based on these first scores, and a judgment result based on the second threshold value Th2. In the judgment result, "True" means that the second score T H，H0 (I) is less than the second threshold value Th2 (i.e., the holding method needs to be switched), "False" means the second score T H，H0 (I) Greater than the second threshold Th2 (ie, no need to switch the holding method).

[0130] [Table 1]

[0131]

[0132] [Table 2]

[0133]

[0134] [Table 3]

[0135]

[0136] In the example shown in Table 1, the second score T H，H0 The average value of objects 1 to 5 of (I) is 1.03, which is greater than 1. Therefore, it can be considered that the holding ease based on the currently selected holding method H0 (clamping) is greater than the holding ease based on the holding method (adsorption) that is the evaluation object on average. H，H0 Since (I) and its average value both exceed the second threshold value Th2 = 0.5, the determination unit 306 of the control device 12 determines that there is no need to switch the holding method.

[0137] Next, the retention strategy determination unit 307 sets the retention order to the first score S H0(I) The holding strategy is determined in descending order by holding the objects in the order of "object 3 → object 2 and object 4 → object 1 and object 5" using the currently selected holding method, i.e., "clamping". Next, the control device 12 stores the determined holding strategy in the storage unit 302. Next, the holding strategy determination unit 307 determines the first holding action in the holding strategy, i.e., "holding object 3 by clamping", as the next holding action. In the example of Table 1, the first score S of "clamping" among two or more objects is H0 When (I) and (I) are the same value, which one is prioritized in the holding strategy can be appropriately determined by taking into account arbitrary factors such as the position and weight of the object O.

[0138] In the example shown in Table 2, the second score T H，H0 The average value of objects 1 to 5 of (I) is 0.79, which is lower than 1. Therefore, it can be considered that the holding ease based on the currently selected holding method H0 (clamping) is lower than the holding ease based on the holding method (adsorption) as the evaluation object on average. However, the second scores T of objects 1 to 5 are H，H0 Since (I) and its average value both exceed the second threshold value Th2 = 0.5, the determination unit 306 of the control device 12 determines that there is no need to switch the holding method.

[0139] Next, similarly to the case of Table 1, the retention strategy determination unit 307 sets the retention order to the first score S H0 (I) The holding strategy is determined so that the currently selected holding method, "clamping," is used to hold the objects in the order of "object 5 → object 1 → object 2 → objects 3 and 4," from largest to smallest. Next, the control device 12 stores the determined holding strategy in the storage unit 302. The holding strategy determination unit 307 then determines the first holding action in the holding strategy, "holding object 5 by clamping," as the next holding action.

[0140] In the example shown in Table 3, the second score T H，H0 The average value of objects 1 to 5 of (I) is 0.38, which is lower than 1. Therefore, it can be considered that the holding ease based on the currently selected holding method H0 (clamping) is lower than the holding ease based on the holding method (adsorption) as the evaluation object on average. In addition, the second scores T of objects 1 to 4 are H，H0 (I) and its average value are both lower than the second threshold value Th2 = 0.5, so the determination unit 306 of the control device 12 determines that it is necessary to switch the holding method.

[0141] Next, the holding strategy determination unit 307 compares the first scores S between “clamping” and “adsorption” for each of the objects 1 to 5. H(I) The size of the object. In this example, among all objects 1 to 5, the first score S of "adsorption" is H (I) Exceed the first score S of the currently selected holding method, i.e., "clamping" H0 (I), therefore, the holding strategy determination unit 307 determines the holding strategy by switching the holding method first. Furthermore, the holding strategy determination unit 307 sets the holding order after the holding method is switched to the first score S of the holding method after the switching, i.e., "adsorption". H (I) The holding strategy is determined in the order of "object 1 and object 5 → object 2 → object 3 → object 4" from the largest to the smallest. Next, the control device 12 stores the determined holding strategy in the storage unit 302. Next, the holding strategy determination unit 307 determines the first holding action in the holding strategy, i.e., "holding object 1 by adsorption" or "holding object 5 by adsorption", as the next holding action. Here, the first score S of the adsorption of object 1 and object 5 is H (I) are equal, so which one is given priority can be appropriately determined by taking into account arbitrary factors such as the positions and weights of objects 1 and 5.

[0142] So far, the determination in step S302 that the first score S is greater than the first threshold value Th1 has been described. H The processing flow of the control device 12 in the case of (I). In contrast, in step S302, it is determined that there is no first score S greater than the first threshold value Th1. H In case (I) (step S302: No), it is considered that there is no object O in a state that is sufficiently easy to maintain. Therefore, the control device 12 executes step S310 to plan a configuration change strategy that makes it easy to maintain the object O, instead of planning the maintenance strategy of steps S303 to S309.

[0143] In step S310, the holding obstacle factor estimation unit 308 of the control device 12 estimates the presence of various holding obstacle factors that make it difficult to hold the object O in the extraction source container V1. Examples of holding obstacle factors include a situation where the objects O are densely packed in a portion of the extraction source container V1, a situation where an obstacle exists in front of the object O to be held, a situation where the object O to be held is arranged in an orientation unsuitable for holding, and a situation where the object O to be held has a shape unsuitable for holding.

[0144] The processing of the holding obstacle factor estimation unit 308 is described by taking the holding obstacle factor of densely packed objects O as an example. The holding obstacle factor estimation unit 308 determines the density of the objects O in the extraction source container V1 based on the image data of the extraction source container V1 acquired by the first sensor 11A. For example, the recognition processing unit 301 sets a shielding area R2 for each object O in the image data acquired by the first sensor 11A, calculates the center CO of the shielding area R2, and the centroid FO of the object O (see FIG. 1 ). Figure 8 ). In addition, the recognition processing unit 301 calculates the distance between the centers CO of each shielding area R2 and the distance between the centroids FO of each object O as the distance between each object O. The retention obstacle factor inference unit 308 can infer that the objects O are densely distributed when the calculated distance between each object O (a representative value such as an average value can also be used) is below a specified threshold. For example, the retention obstacle factor inference unit 308 can infer that the objects O are densely distributed when the inner side of a circle with a specified threshold as the radius from the center point CO, FO of a specific object O contains one or more (for example, multiple) center points CO, FO of other objects O. By performing density judgment in this way, the retention obstacle factor inference unit 308 can determine whether there is a retention obstacle factor that objects O are densely distributed.

[0145] In addition to or in lieu of the above method, the recognition processing unit 301 can determine whether the object O (when viewed from above) has an elongated shape based on the aspect ratio of the obstruction region R2 corresponding to the object O. For example, if multiple objects O that are close together are determined to have elongated shapes, the retention-impeding factor inference unit 308 can infer that the objects O, which are thin in thickness, are densely packed vertically. Alternatively, the retention-impeding factor inference unit 308 can infer that the objects O are arranged in an orientation unsuitable for retention based on the vertical orientation of the objects O.

[0146] In addition to the above, the holding obstacle estimation unit 308 can also estimate the presence or absence of various holding obstacles. For example, the holding obstacle estimation unit 308 can estimate the presence of an obstacle that hinders the holding of the object O to be held, or infer that the object O is positioned in a posture that is not suitable for holding, based on the image data from the first sensor 11A.

[0147] Next, the control device 12 executes step S311. In step S311, the configuration change strategy determination unit 309 of the control device 12 determines the holding method and configuration change operation corresponding to the holding obstacle inferred in step S310. For example, if the holding obstacle inferred in step S310 is a densely packed object O, the configuration change strategy determination unit 309 may select an operation to move the object O to eliminate the dense packing as the configuration change operation. The configuration change strategy determination unit 309 may select the specific configuration change operation (e.g., pressing the object O, pulling the object O, pushing the object O, picking up the object O, etc.) based on the density pattern. Furthermore, if the holding obstacle inferred in step S310 is an obstacle located above the object O being held, the configuration change strategy determination unit 309 may select an operation to move the obstacle as the configuration change operation. This association between the holding obstacle, the configuration change operation, and the holding method to be used may be established by reference to an association table or database, may be appropriately specified by the user, or may be learned through machine learning. However, at this stage, the object O to be the target of the configuration change operation is not determined. Figures 10 to 13 Provide explanation. Figures 10 to 13 This is a schematic diagram showing an example of a configuration change operation.

[0148] For example, Figure 10 As shown, when five elongated objects O are densely packed in one corner of the extraction source container V1, the configuration change strategy determination unit 309 can select, as a configuration change operation targeting the second object O from the left in the figure, a configuration change operation that moves the object O in the direction from the centroid Go of the object O toward the centroid Gs of the region S where no object O exists. Here, the recognition processing unit 301 can convert the depth image D (see Figure 6 ) is set as an area S where the object O does not exist.

[0149] Furthermore, when setting the area S where the object O does not exist, the recognition processing unit 301 can also search for divided areas obtained by dividing the internal area of ​​the extraction source container V1 into a plurality of areas, instead of searching the entire internal area of ​​the extraction source container V1. Figure 11 As shown, the recognition processing unit 301 can also use a dividing line shown by a single dot-dashed line to divide the entire extraction source container V1 into two parts, and search for areas where the object O does not exist in the upper half and the lower half respectively. Figure 11The area S1 is set as an area where the object O does not exist, and the configuration change strategy determination unit 309 can select the configuration change operation of moving the object O in the direction from the centroid Go of the object O toward the centroid Gs1 of the area S1. Figure 11 Region S2 is set as an area where object O does not exist, and the configuration change strategy determination unit 309 can select a configuration change operation to move object O in the direction from the centroid Go of object O toward the centroid Gs2 of region S2. This segmentation process allows the movement direction selected as a configuration change operation to be limited to a desired specific region, and the selection of configuration change operations can be adjusted as needed. Furthermore, the region segmentation method is not limited to the above example; any method can be used, such as dividing the region into two parts or four parts.

[0150] The configuration change strategy decision unit 309 may also select not only the configuration change operation but also the holding method based on the inferred holding obstacle. For example, the configuration change strategy decision unit 309 may decide to use the first holding unit 200A to perform the configuration change when it is inferred that there is a holding obstacle that is effective for the configuration change based on "clamping". In addition, the configuration change strategy decision unit 309 may decide to use the second holding unit 200B to perform the configuration change when it is inferred that there is a holding obstacle that is effective for the configuration change based on "adsorption". In this way, the configuration change can be efficiently performed using the holding unit that is most suitable for the determined configuration change operation. However, the configuration change strategy decision unit 309 may also select the currently selected holding unit 200 as the holding method regardless of the inferred holding obstacle, and determine the configuration change operation based on the use of the currently selected holding unit 200. In this case, the number of switching times of the holding unit is reduced, thereby improving the throughput of the conveying system 1.

[0151] Generally, when attempting to hold the object O by "clamping", it is effective to change the configuration so as to increase the distance between the objects O. On the other hand, when attempting to hold the object O by "adsorption", it is effective to change the configuration so as to increase the area of ​​the region that can adsorb the object O. For example, Figure 12As shown, the configuration change operation of the object O by "gripping" can be performed by inserting the front end of the clamping manipulator 202 between the two objects O and opening the clamping manipulator 202 to increase the distance between the objects O. In addition, the following various configuration change operations can be selected: a method of increasing the distance between the objects O by moving the clamping manipulator 202 in one direction or rotating the root axis (six axes in the case of a multi-joint robot) of the clamping manipulator 202 after inserting the front end of the clamping manipulator 202 between the two objects O; a method of overturning the object O by moving the clamping manipulator 202 in the lateral direction while the front end of the clamping manipulator 202 is in contact with the upper surface of the object O; a method of changing the configuration by slightly lifting a specific object O by the clamping manipulator 202 and then dropping it; a method of lifting a specific object O by the clamping manipulator 202 and dropping it from other objects O; a method of moving in a manner of sweeping over other objects O while clamping a specific object O (for example, dragging the object O while holding the end of the object O); and the like. Furthermore, the gripping robot 202 can also grip the object O by flattening the convex and concave surfaces thereof, or by pressing the object O against a wall to deform the object O.

[0152] The configuration change operation of object O based on "adsorption" is as follows Figure 13 As shown, the object O can be overturned by moving the suction device 203 in the lateral direction while the suction portion 205 of the suction device 203 is in contact with the upper surface of the object O. In addition, various configuration change operations can be selected, such as a method of rotating the suction device 203 while the suction portion 205 is in contact with the object O, a method of lifting a specific object O with the suction device 203 and dropping it from other objects O, and a method of moving the suction device 203 so as to sweep over other objects O while adsorbing a specific object O.

[0153] Next, the control device 12 executes step S312. In step S312, the score calculation unit 304 calculates a "third score" as an indicator of the effectiveness of the relocation operation. The score calculation unit 304 calculates the third score for each object O or region of object O based on the retention obstacle factor inferred in step S310 and the retention method and relocation operation determined in step S311. For example, if the retention obstacle factor of densely populated objects O is inferred in step S310, and the relocation operation of "moving one object O toward a region S where no objects O exist" is determined in step S311 while "clamping" is selected as the retention method, the score calculation unit 304 calculates a third score indicating the effectiveness of this relocation operation for each object O or region of object O that can be retained. By comparing these third scores, it is possible to select a preferred object O or region of object O as the target of the relocation operation.

[0154] The score calculation unit 304 can calculate the third score based on, for example, the ease of the relocation operation for the target object O, the difficulty of maintaining the target object O, and the number of other objects O predicted to be relocated following the relocation operation for the target object O. The easier the relocation operation for the target object O is, the higher the effectiveness of the relocation operation can be evaluated. Since it is assumed that the more difficult the target object O is to maintain, the easier it is to maintain it through the relocation operation, the higher the effectiveness of the relocation operation can be evaluated. The relationship between the number of objects O relocated together in a single relocation operation and the effectiveness of the relocation operation varies depending on the size of the object O, etc. Specifically, there is an appropriate number of objects O relocated together in a single relocation operation, depending on the size of the object O, etc., and the closer the number is to this number, the higher the effectiveness of the relocation operation can be evaluated. For example, if the object O is small, the greater the number of objects O relocated together, the easier it is to maintain the relocation, and the higher the effectiveness of the relocation operation. On the other hand, if the object O is large, maintaining the relocation may become difficult if too many objects O are relocated together.

[0155] The ease of the relocation operation for object O can be evaluated similarly to the estimation of retention obstacles in step S310, for example, based on the placement, shape, and density of objects O. The difficulty of retaining object O can be evaluated, for example, based on the same evaluation criteria as the first score representing the ease of retaining, with a negative correlation to the first score. The number of other objects O predicted to be relocated following the relocation operation for the target object O can be evaluated based on, for example, the placement of each object O and the details of the relocation operation.

[0156] The score calculation unit 304 can calculate the third score using an evaluation function represented by a linear combination of the evaluation items. However, the evaluation function is not limited to the above example; any evaluation function, such as a nonlinear function, can be used. Furthermore, the evaluation items are not limited to the above example; for example, they may be quantities that depend on the characteristics of the object O, the characteristics of the holding unit 200 used, the characteristics of the selected configuration change operation, and the like.

[0157] Next, the control device 12 executes step S313. In step S313, the configuration change strategy determination unit 309 of the control device 12 determines a configuration change strategy based on the calculated third score. For example, the configuration change strategy determination unit 309 may determine to perform the configuration change operation determined in step S311 on the object O or the region of object O with the highest third score. Alternatively, the configuration change strategy determination unit 309 may determine the order of configuration changes by sorting the objects O or regions of object O in descending order of their third scores, similar to the retention strategy.

[0158] Next, the control device 12 executes step S314. In step S314, the control device 12 stores the arrangement change policy determined by the arrangement change policy determination unit 309 in step S313 in the storage unit 302. As described above, the control device 12 can create an arrangement change policy for the object O when it is determined that the object O is difficult to hold.

[0159] In the above example, the third score is calculated in step S312 based on the holding method and placement change operation determined in step S311. However, the calculation of the third score is not limited to this. For example, in step S312, the score calculation unit 304 can calculate the third score for each object O or region of object O for multiple holding methods and / or multiple change operations. For example, if there are 10 objects O or regions of object O for score calculation, and the third score is calculated for each of the two holding methods, "clamping" and "adsorption," a total of 10 × 2 = 20 third scores will be calculated. Furthermore, for example, in the case where five configuration change operations are predefined, namely, (1) "inserting the front end of the holding unit 200 between objects O and opening it," (2) "inserting the front end of the holding unit 200 between objects O and moving it in one direction," (3) "inserting the front end of the holding unit 200 between objects O and rotating it," (4) "moving the front end of the holding unit 200 laterally while in contact with the upper surface of the object O," and (5) "lifting the object O and dropping it," the control device 12 can select one or more of the above configuration change operations as the target for score calculation based on various information obtained by the sensor 11, and calculate the third score for each selected configuration change operation. By selecting the configuration change operation as the target for score calculation according to the situation, the calculation time and load can be reduced compared to performing all the specified configuration change operations. For example, in the case where there are 10 objects O or regions of the object O as the target for score calculation, when two configuration change operations are selected as candidates, a total of 10×2=20 third scores are calculated. By selecting the largest score among the third scores thus calculated, not only can the object O or region of the object O to be retained be determined based on the third score, but the retention method and configuration change operation to be adopted can also be determined based on the third score. In this case, steps S310 and S311 can be omitted. Alternatively, the control device 12 can omit the selection of the configuration change operation and calculate the third score for all types of configuration change operations.

[0160] As an example of a method for selecting an appropriate configuration change operation as described above, the control device 12 can use, for example, the evaluation function g(I) shown in the following equation 6. g(I) represents the third score of the configuration change operation for the shielding area I showing the object O or the area of ​​the object O.

[0161] [Formula 6]

[0162] g(I)=argmax(w1f1,w2f2,w3f3,w4f4,w5f5)

[0163] In Formula 6, f1 represents the softness of object O. f2 represents the surface area of ​​object O that is predicted not to overlap (i.e., be exposed) with other objects O after the configuration change. f3 represents the aspect ratio of object O. f4 represents the height difference between object O and its surroundings (inner and outer heights). f5 represents the spatial margin, which indicates the size of the space around object O where no other objects O exist. w1 to w5 represent the weights of each function.

[0164] The relationship between the parameters f1 to f5 of the evaluation function g(I) and the above-mentioned configuration change operations (1) to (5) can be expressed, for example, as shown in the following table. For example, when the f1 component is the largest in the evaluation function g(I), (1) and (3) are obtained as candidates for the configuration change operation. In this case, since there are multiple candidates, the control device 12 confirms the largest parameter next to f1. Specifically, the control device 12 compares the parameters f3 and f4 corresponding to (1) or (3), selects the configuration change operation (3) when the f3 component is greater than the f4 component in the evaluation function g(I), and selects the configuration change operation (1) when the f3 component is less than the f4 component. In addition, the control device 12 can also omit the comparison process of f3 and f4, calculate the third score for both the configuration change operations (1) and (3), and select the configuration change operation with the larger third score.

[0165] [Table 4]

[0166] parameter Evaluation Project Configuration change operations <![CDATA[f1]]> Softness (1)(3) <![CDATA[f2]]> Surface area during configuration changes (2)(4) <![CDATA[f3]]> Aspect ratio (3)(4) <![CDATA[f4]]> Internal and external height (1)(2) <![CDATA[f5]]> Surrounding space affluence (4)

[0167] The control device 12 may also perform the above-described configuration change operation selection process instead of steps S310 and S311. In this case, the control device 12 can select an appropriate type of configuration change operation without specifically estimating the retention inhibitor, thereby reducing computational cost.

[0168] Refer again Figure 3 Next, the control device 12 executes step S4. In step S4, the determination unit 306 of the control device 12 determines whether a configuration change is necessary. If the determination unit 306 determines that there is no first score greater than the first threshold Th1 (step S302: No), the control device 12 determines that a configuration change is necessary. If the determination unit 306 determines that there is a first score greater than the first threshold Th1 (step S302: Yes), the control device 12 determines that a configuration change is not necessary. If the determination unit 306 determines that a configuration change is necessary (step S4: Yes), the control device 12 executes step S5.

[0169] In step S5, the motion control unit 303 of the control device 12 specifically calculates the position at which the relocation operation of the target object O is to be performed, as well as the posture of the movable arm 100 used to perform the relocation operation, based on the relocation strategy determined in step S313. Based on the calculation results, the motion control unit 303 controls the movable arm 100 and the holding unit 200 to perform the relocation operation of the object O (relocation operation step). Specifically, the relocation operation is performed on the object O with the highest third score, so that the holding unit 200 can easily hold the object O. The control device 12 then returns to step S2, where the first sensor 11A captures the state of the removal source container V1 after the relocation as image data. Next, the first score is calculated again in step S301, and each first score is compared with the first threshold Th1 in step S302. This relocation operation is repeated until the first score of one or more objects O exceeds the first threshold Th1.

[0170] On the other hand, if the determination unit 306 determines that a configuration change is not necessary (step S4: No), the control device 12 executes step S6. In step S6, the motion control unit 303 of the control device 12 specifically calculates the position of the holding target object O determined in step S3 and the posture of the movable arm 100 during holding (holding position and posture planning step).

[0171] When the holding method of the object O selected in step S3 is "clamping", in step S6, the action control unit 303 of the control device 12 specifically calculates the position of the clamped object O and the posture of the movable arm 100 during clamping by a method appropriately selected from known methods.

[0172] When the holding method of the object O selected in step S3 is "adsorption", in step S6, the motion control unit 303 of the control device 12 specifically calculates the position of the adsorbed object O and the posture of the movable arm 100 during adsorption by a method appropriately selected from known methods.

[0173] In addition, in this embodiment, the selectable holding method is either "clamping" or "adsorption", but for example, in the case where "clamping and adsorption (mixed)" can be used, in step S6, the motion control unit 303 of the control device 12 can specifically calculate the position of the clamped and adsorbed object O and the posture of the movable arm 100 during clamping and adsorption by a method appropriately selected from known methods.

[0174] After calculating the position of the object O to be held and the posture of the movable arm 100 during holding, the motion control unit 303 calculates whether the calculated holding action can be used as an actual robot action. If it is determined that it cannot be used as an actual action, the position of the object O to be held and the posture of the movable arm 100 during holding are recalculated. If it is determined that it cannot be used as an actual action, the process may return to step S3 and execute again from the planning of the holding strategy.

[0175] The calculation of the position of the held object O and the posture of the movable arm 100 requires a significant amount of computation. The control device 12 only needs to calculate the position of the held object O and the posture of the movable arm 100 for the selected object O. Therefore, the conveying system 1 can significantly reduce the required computational effort compared to other conveying systems that calculate the position of the held object O and the posture of the movable arm 100 in order to select the held object O.

[0176] Next, the control device 12 executes step S7. In step S7, the determination unit 306 of the control device 12 determines whether the holding method needs to be switched. If the determination unit 306 determines that the holding method needs to be switched (step S7: Yes), that is, if the currently selected holding method is different from the holding method in the next holding operation, the control device 12 executes step S8. In step S8, the motion control unit 303 switches the holding method (holding method switching process). For example, when switching from "gripping" to "adsorption", the motion control unit 303 controls the movable arm 100 of the transport device 10 so that the second holding unit 200B is directed toward the removal source container V1 instead of the first holding unit 200A directed toward the removal source container V1. On the other hand, if the determination unit 306 determines that the holding method does not need to be switched (step S7: No), that is, if the currently selected holding method is the same as the holding method in the next holding operation, the control device 12 does not execute step S8 and proceeds to step S9.

[0177] Next, the control device 12 executes step S9. In step S9, the control device 12 controls the holding unit 200 and the movable arm 100 based on the position of the held object O and the posture of the movable arm 100 calculated in step S6 (operation control step). The selected object O is then transported from the removal source container V1 to the transport destination container V2 by the transport device 10.

[0178] Next, the control device 12 executes step S10. In step S10, the control device 12 determines whether the target object O has been successfully held and / or conveyed based on, for example, image data acquired by the first sensor 11A and the second sensor 11B (holding success determination step). The control device 12 stores the determination result in the storage unit 302.

[0179] Next, the control device 12 executes step S11. In step S11, the judgment unit 306 of the control device 12 determines whether the instructions recorded in the instruction list have been completed. For example, the judgment unit 306 determines whether there is an object O remaining in the extraction source container V1. When the judgment unit 306 determines that there is an object O remaining in the extraction source container V1 (step S11: No), the control device 12 executes step S2 again. That is, the control device 12 again obtains information related to the state of the extraction source container V1 after picking up the object O, the state of each holding unit 200A, 200B, etc., and makes the next holding strategy based on this. In addition, the control device 12 may not perform the holding strategy planning process, but may determine the next holding action based on the holding strategy including the order of the holding that has been produced. In addition, the control device 12 may also correct and update the already produced holding strategy based on the acquired information instead of performing the same holding strategy planning process. Furthermore, the control device 12 may cause the score calculation unit 304 to change the value of the parameter used when generating the score, or cause the threshold generation unit 305 to change the first threshold Th1 and / or the second threshold Th2 after the holding operation is performed once or more.

[0180] On the other hand, if the determination unit 306 determines that the instructions listed in the instruction list have been completed (step S11: Yes), the control device 12 executes step S9 and ends the control. For example, if the determination unit 306 determines that no object O remains in the removal source container V1, the control device 12 executes step S9 and ends the control.

[0181] According to the structure of the first embodiment described above, when objects are difficult to hold, the effectiveness of the relocation operation is evaluated for each object or region of the object that can be subjected to the relocation operation, thereby improving the ease of holding caused by the relocation operation. This can shorten the operation time of the transport device and improve the throughput of the system.

[0182] Furthermore, in this embodiment, the control device 12 calculates a configuration change operation score (third score) for each object O during the evaluation, and selects an object O as a target for configuration change based on the score. The configuration change operation score calculated for each object O is calculated based on at least one of the ease of configuration change operation of the target object O, the difficulty of retaining the target object O, and the number of objects O predicted to undergo configuration change in conjunction with the configuration change operation of the target object O. Therefore, the object O predicted to be most effectively affected by the configuration change operation can be selected as the target for the configuration change operation.

[0183] Furthermore, in this embodiment, the control device 12 infers a retention-impeding factor that hinders retention of the object O based on information acquired from the sensor 11, and determines a relocation operation based on the inferred retention-impeding factor and the evaluation result. This allows the relocation method to be determined in accordance with the factor currently hindering the retention operation, thereby improving the effectiveness of the relocation operation.

[0184] Furthermore, in this embodiment, the control device 12 determines the density of the objects O based on at least one of the arrangement and shape of the objects O, and determines the arrangement change operation so as to reduce the density of the objects O. Thus, in a case where the objects O are densely packed and difficult to maintain, the arrangement change can be performed so as to eliminate the dense density of the objects O.

[0185] Furthermore, in this embodiment, the holding unit 200 is configured to hold the object O by at least one of clamping and adsorption. When the holding unit 200 holds the object O by clamping, the control device 12 determines the arrangement change operation so that the distance between the objects O increases. When the holding unit 200 holds the object O by adsorption, the control device 12 determines the arrangement change operation so that the area capable of adsorbing the object O increases. Thus, an appropriate arrangement change operation can be selected depending on the type of holding unit.

[0186] Furthermore, in this embodiment, the holding unit 200 can select one or more of a plurality of holding methods to hold the object O. Based on the evaluation results, the control device 12 determines the object O to be subjected to the placement change operation and the holding method of the holding unit 200 to be used for the placement change operation. This allows selection of a holding unit that facilitates efficient placement change operations.

[0187] Furthermore, in this embodiment, the holding unit 200 can select one or more of a plurality of holding methods to hold the object O, and the holding method currently selected by the holding unit 200 is used as the holding method of the holding unit 200 used in the configuration change operation. This avoids changing the currently selected holding method, and improves the throughput of the system.

[0188] Furthermore, in this embodiment, the control device 12 calculates a holding ease score (first score) for each object O based on information acquired from the sensor 11, and determines whether to reposition the object O based on the holding ease score. If there is no object O with a holding ease score that is equal to or greater than a predetermined threshold (first threshold), the control device 12 determines to reposition the object O. Thus, if there is an object O with a predetermined holding ease, the holding operation is prioritized over the repositioning operation, thereby improving system throughput.

[0189] Furthermore, in this embodiment, the control device 12 selects one or more configuration change operations from a plurality of configuration change operations based on at least one of information about the object O and information acquired from the sensor 11, and evaluates the effectiveness of the selected configuration change operations for each object O. The control device 12 selects one or more configuration change operations from the plurality of configuration change operations based on one or more pieces of information selected from the group consisting of the flexibility of the object O, the surface area of ​​a portion of the object O predicted to be exposed after the configuration change operation, the aspect ratio of the object O, the height difference between the object O and its surroundings, and the size of the space surrounding the object O. This significantly reduces computational cost compared to calculating the third score for each object O for all types of configuration change operations.

[0190] Furthermore, in this embodiment, the control device 12 calculates the position at which the reconfiguration operation is to be performed and the posture of the movable arm 100 used to perform the reconfiguration operation. This allows the control device 12 to create a reconfiguration strategy without specifically calculating the position at which the reconfiguration operation is to be performed, the posture of the movable arm 100, and so on. The control device 12 only needs to calculate the position at which the reconfiguration operation is to be performed, the posture of the movable arm 100, and so on, for the object O ultimately selected as the reconfiguration target, significantly reducing the required computational effort.

[0191] In addition, in this embodiment, the conveying device 10 has a movable arm 100, a holding portion 200, a sensor 11, and a control device 12. The holding portion 200 is mounted on the movable arm 100 and is capable of selecting one or more holding methods to hold the object O. The sensor 11 is capable of detecting multiple objects O. The control device 12 controls the movable arm 100 and the control device 12. Based on the information obtained from the sensor 11, the control device 12 calculates a score (first score) based on the selected holding method for each object O and each holding method. The control device 12 selects the object O and holding method to be held next based on the score. The control device 12 calculates the position at which the selected object O is held and the posture of the movable arm 100. Thus, it is possible to efficiently determine the holding strategy for holding the object O while suppressing the number of times the holding method is switched. The control device 12 selects the holding strategy without specifically calculating the position at which the object O is held, the posture of the movable arm 100, etc. The holding strategy refers to, for example, the selection of the type of conveying device 10 used, the object O held, and the holding method. The control device 12 only needs to calculate the position of the object O to be held and the posture of the movable arm 100 for the selected object O, which can significantly reduce the required amount of calculations. In addition, if the currently selected holding method is relatively easy to hold, the holding strategy can be created so that the currently selected holding method is prioritized without switching the holding method. This can save the time required to switch the holding method, thereby shortening the overall operation time.

[0192] In this embodiment, the plurality of holding methods include clamping and suction. Thus, the transport device 10 can hold various objects O, such as thin objects O that are difficult to hold by clamping or objects O with concave and convex shapes that are difficult to hold by suction.

[0193] Furthermore, in this embodiment, the control device 12 selects the object O to be held and the holding method based on the score, so as to minimize the number of times the holding method needs to be switched. Furthermore, the control device 12 determines whether a holding method switch is necessary (or necessary) based on the score. This allows the transport device 10 to perform a series of holding operations while minimizing the number of times the holding method needs to be switched, thereby reducing the time required for the overall operation.

[0194] Furthermore, in this embodiment, the control device 12 determines whether or not a change in the retention method is necessary based on the magnitude relationship between the score and a predetermined second threshold value Th2. This allows the control device 12 to easily determine the necessity of a change in the retention method. Furthermore, the control device 12 can create a retention strategy tailored to the situation by adjusting the second threshold value Th2.

[0195] Furthermore, in the present embodiment, the control device 12 determines the order of holding the objects O based on the scores. This allows the control device 12 to select the next holding operation in consideration of the entire picking operation specified in the instruction list.

[0196] In addition, in this embodiment, the control device 12 determines the order of switching the holding method of the object. This allows the control device 12 to select the optimal timing for switching the holding method in consideration of the entire picking operation specified in the instruction list.

[0197] Furthermore, in this embodiment, when calculating the score for the case where the holding method is suction, the control device 12 calculates the score for each region of the object O. This allows the control device 12 to select the region that is easiest to hold for the same object O as the holding target region, thereby improving holding efficiency.

[0198] Furthermore, in this embodiment, the control device 12 calculates the ease of holding for each object O and each holding method as a first score, and the score based on the selected holding method is a second score calculated for each object O and each holding method based on the first score. Thus, the control device 12 can determine the holding action based on two perspectives: the ease of holding the object O represented by the first score, and the priority level based on the currently selected holding method represented by the second score.

[0199] In addition, in this embodiment, the second score for the object O and the holding method is a value obtained by dividing the first score for the object O or the region of the object O and the selected holding method by the first score for the object O or the region of the object O and the holding method. This allows the control device 12 to easily calculate the second score for determining the necessity of switching the holding method.

[0200] (Second embodiment)

[0201] Next, refer to Figure 14 The second embodiment will be described. The second embodiment differs from the first embodiment in that the score calculation is corrected according to the success or failure of the hold. Except for the configuration described below, the other configurations are the same as those of the first embodiment.

[0202] Figure 14 This is a control flowchart of a maintenance strategy planning process performed by the control device 12 according to the second embodiment.

[0203] In the second embodiment, the control device 12 executes steps S11 and S12 after step S10 (holding success / failure determination step) of the first embodiment. As in the first embodiment, the control device 12 plans a holding strategy or a configuration change strategy based on the instruction list and information acquired by the sensor 11, controls the movable arm 100 and the holding unit 200 so as to perform holding or configuration change based on the planned strategy, and, if holding is successful, determines in step S10 whether the object O was successfully held.

[0204] After step S10, if the determination unit 306 determines that retention was successful (step S11: Yes), the control unit 12 determines whether the instruction list is complete (step S13), similar to the first embodiment. On the other hand, if the determination unit 306 determines that retention failed (step S11: No), the control device 12 executes step S12. In step S12, the control device 12, for example, refers to the retention policy stored in the storage unit 302 in step S308 and identifies the area where the object O that failed to be retained is located as a retention failure area. The score calculation unit 304 adjusts the score so that the first score of the object O included in the determined retention failure area is reduced. Alternatively, or in addition to the above example, the threshold generation unit 305 may adjust the first threshold Th1.

[0205] The failed retention area can be set to include only the object O that failed to be retained, or it can be set to an area of ​​any shape, such as a circle or polygon, of a predetermined size, centered on the failed retention object O. Alternatively, the internal area of ​​the extraction source container V1 can be pre-divided into multiple areas, and the area containing the failed retention object O in each divided area can be set as the failed retention area. However, the method for setting the failed retention area is not limited to the above example, and any method can be used.

[0206] The score calculation unit 304 can perform the correction process by, for example, multiplying the first score calculated in the usual manner by a correction factor between 0 and 1 when subsequently calculating the first score for the object O included in the failed retention region, or by correcting the parameters of the evaluation function used to calculate the score for the object O included in the failed retention region. However, the method for correcting the first score is not limited to the above example; any method can be used. Furthermore, the correction process can be performed in another step, such as during the calculation of the second score, instead of the first score, to correct the priority of the object O in the retention strategy.

[0207] Next, the control device 12 determines whether the instruction list is completed in the same manner as in the first embodiment (step S13 ).

[0208] According to the configuration of this embodiment, the control device 12 calculates a score (a first score or a second score) for each object O, used to determine the priority of the holding operation, based on information acquired from the sensor 11. If the holding unit 200 determines that the holding operation has failed, the control device 12 adjusts the priority score based on information about the object O for which the holding operation failed. This allows the control device 12 to appropriately adjust the priority of each object O in the holding strategy based on the success or failure of the holding operation. As a result, it is possible to avoid situations where the holding unit 200 repeatedly attempts to hold an object O that is difficult to hold, thereby improving the success rate of the holding operation in the holding strategy.

[0209] (Third embodiment)

[0210] Next, refer to Figure 15 The third embodiment will be described. The third embodiment differs from the first embodiment in that a plurality of retention strategies are created. Except for the configuration described below, the other configurations are the same as those of the first embodiment.

[0211] Figure 15 This is a control flowchart of a maintenance strategy planning process performed by the control device 12 according to the third embodiment.

[0212] In the third embodiment, the control device 12 executes steps S357 to S359 instead of step S307 of the first embodiment. In the maintenance strategy planning step (step S3), the control device 12 calculates the first score S in the same manner as in the first embodiment. H (I) (Step S351), when there is no first score S greater than the first threshold Th1 H In the case of (I) (step S352: No), the holding obstacle is estimated in the same manner as in the first embodiment (step S362), the holding method and the configuration change operation are determined (step S363), the third score is calculated for each object O or the region of the object O (step S364), the configuration change strategy is determined (step S365), and the configuration change strategy is stored in the storage unit 302 (step S366). On the other hand, when there is a first score S greater than the first threshold value Th1, H In the case of (I) (step S352: Yes), the control device 12 calculates the second score T H，H0(I) (Step S353), the necessity of switching the holding method is determined (Step S354). If the determination unit 306 determines that the holding method does not need to be switched (S355: No), the holding strategy determination unit 307 of the control device 12 determines a holding strategy including a holding order, similar to the first embodiment (Step S356), stores the holding strategy in the storage unit 302 (Step S360), and determines the next holding action based on the holding strategy (Step S361). In other words, the control content of the control device 12 when the determination unit 306 determines that the holding method does not need to be switched is the same as in the first embodiment.

[0213] On the other hand, if the determination unit 306 determines that the holding method needs to be switched (S355: Yes), the control device 12 executes step S357. In step S357, the holding strategy determination unit 307 of the control device 12 creates a plurality of holding strategy plans including the order of holding and switching. For example, the holding strategy determination unit 307 determines the holding strategy based on the first score S H (I) Create a holding strategy plan that performs one switch and has a relatively low overall success rate, and a holding strategy plan that performs two switches and has a relatively high overall success rate.

[0214] Next, the control device 12 executes step S358. In step S358, the score calculation unit 304 of the control device 12 calculates the score of each holding strategy plan created by the holding strategy determination unit 307. For example, the score calculation unit 304 calculates the score of each holding strategy plan based on the number of switching times of the holding method in the holding strategy plan, the first score S of each holding action, and the number of switching times of the holding method in the holding strategy plan. H The score of each maintenance strategy plan is calculated using the average value, maximum value, minimum value, etc. of (I), the predicted time required to complete the maintenance strategy, etc. as evaluation items. The score calculation unit 304 can also appropriately change the weight of the evaluation items based on the operator's input, for example.

[0215] Next, the control device 12 executes step S359. In step S359, the retention strategy determination unit 307 of the control device 12 determines the optimal retention strategy based on the scores of the retention strategy proposals generated by the score calculation unit 304. For example, the retention strategy determination unit 307 selects the retention strategy proposal with the highest score as the retention strategy.

[0216] Then, the control device 12 stores the holding strategy determined by the holding strategy determination unit 307 in the storage unit 302 (step S360 ), and determines the next holding operation based on the holding strategy (step S361 ).

[0217] Furthermore, the control device 12 may omit the calculation of the second score and / or the determination of the necessity of switching (steps S353-S355) and instead create multiple retention strategy proposals based on the first score. For example, the retention strategy determination unit 307 may create a retention strategy proposal with the lowest overall success rate, which does not require switching; a retention strategy proposal with the second lowest overall success rate, which requires switching once; and a retention strategy proposal with the highest overall success rate, which requires switching twice. The score calculation unit 304 calculates the scores for each retention strategy proposal, and the retention strategy determination unit 307 determines the retention strategy based on the scores. Thus, even without determining whether to switch the retention method, the control device 12 can determine a retention strategy that reduces the number of retention method switches.

[0218] According to the configuration of this embodiment, the control device 12 creates multiple holding strategies that include at least one of the order in which the object O is held and the order in which the holding method is switched, and selects one of the multiple holding strategies. This allows the control device 12 to compare and study a wide range of holding strategies and select the holding strategy that excels in terms of success rate, number of holding method switches, and so on.

[0219] In addition, in this embodiment, the control device 12 selects one of the multiple holding strategies so as to minimize the number of switching holding methods. As a result, the transport device 10 can perform a series of holding operations so as to minimize the number of switching holding methods, thereby reducing the time required for the overall operation.

[0220] (Fourth embodiment)

[0221] Next, the fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that a holding method is selected based on the frequency of use of each holding method. Except for the configuration described below, the other configurations are the same as those of the first embodiment.

[0222] The control device 12 records the usage frequency of each holding method in the storage unit 302. For example, the control device 12 can record the holding method used in the storage unit 302 each time a holding action is performed, and calculate the usage frequency of each holding method based on the usage history of the holding method. Alternatively, the control device 12 can record the switching of the holding method in the storage unit 302 and calculate the usage frequency of each holding method based on the switching history of the holding method. In this way, the control device 12 can confirm the frequency with which each holding method has been used in the past.

[0223] The control device 12 can reduce the frequency of use of the frequently used holding method when there is a deviation in the calculated frequency of use of each holding method or when it is determined that the holding method is approaching the usage limit. Figure 9 In step S303, the second score T is calculated. H，H0 (I) When adjusting the second score T H，H0 (I) in order to reduce the priority of selecting the frequently used holding method. Alternatively, the control device 12 can be, for example, Figure 9 In step S304, the second score T H，H0 (I) When compared with the second threshold value Th2, the threshold value generating unit 305 adjusts the second threshold value Th2 to reduce the priority of selecting the frequently used holding method. Alternatively, the control device 12 can, for example, Figure 9 In step S307, when determining a holding strategy including the order of holding and switching, in order to avoid using a frequently used holding method as much as possible, the first score S based on other holding methods is H (I) A relatively high object O is held by the other holding method. Alternatively, the control device 12 can, for example, Figure 15 In step S358, when calculating the scores of the various retention strategy solutions, the parameters in the scores are adjusted to reduce the priority of selecting the frequently used retention methods.

[0224] For example, the control device 12 can calculate the fractions of clamping and adsorption using the following formula 7.

[0225] [Formula 7]

[0226] F(I)=g(w1 h1S p (I)+w2h2S s (I))+w3f item

[0227] In formula 7, w1h1+w2h2+w3=1, w i ≥0(i=1, 2, 3), h j ≥0(j=1,2). S p (I) is the first score when the occlusion region I representing the object O or the region of the object O is held by "clamping". S p (I) is the first score when the occlusion region I representing the object O or the region of the object O is held by "adsorption". g(S H (I), S H0 (I)) is a function used to calculate the second score. item The weight of the object O is multiplied by the coefficient and normalized. h1 and h2 are weights for adjusting the degree of utilization of the current holding method. For example, h1 and h2 can be stored in a database as table information as shown below. In addition, the control device 12 can also be based on the first score S p (I) or S s(I) to adjust the weight.

[0228] [Table 5]

[0229] Maintenance method <![CDATA[h1]]> <![CDATA[h2]]> adsorption 0.9 0.1 Clamping 0.1 0.9

[0230] According to the structure of the fourth embodiment described above, the control device 12 selects the object O to be held and the holding method based on the frequency of use of each holding method. Thus, the control device 12 can suppress the preference for a particular holding method by adjusting the priority of the frequently used holding method, thereby extending the life of the holding unit 200.

[0231] Furthermore, in this embodiment, the control device 12 calculates a score based on the frequency of use of each holding method. Alternatively, the control device 12 determines a second threshold value Th2 based on the frequency of use of each holding method and determines whether a holding method switch is necessary (or necessary) based on the relationship between the score and the second threshold value Th2. Thus, by modifying the score or threshold value based on the frequency of use of each holding method, the control device 12 can easily perform adjustments such as lowering the priority of frequently used holding methods.

[0232] (Fifth embodiment)

[0233] Next, the fifth embodiment will be described. The fifth embodiment differs from the first embodiment in that a holding method is selected based on the detection results of the physical states of the holding units 200A and 200B. Except for the configuration described below, the other configurations are the same as those of the first embodiment.

[0234] The control device 12 receives information related to the physical state of the first holding portion 200A and the second holding portion 200B from the third sensor 11C and the fourth sensor 11D. If a specific holding portion 200 is used repeatedly, the measurement values ​​of the physical sensors of the third sensor 11C and the fourth sensor 11D may change due to deformation, shape change, surface condition change, etc. of the holding portion 200. Based on this information related to the physical state, the control device 12 infers the usage state, such as the wear state, of the holding portions 200A and 200B. Regarding the holding method using the holding portions 200A and 200B that are inferred to have deteriorated or damaged, the control device 12 adjusts the score, threshold, and the order of holding and switching in the holding strategy, etc., in the same manner as in the fourth embodiment, to reduce the priority or frequency of selecting the holding method.

[0235] According to the configuration of the fifth embodiment described above, the control device 12 selects the object O to be held and the holding method based on the physical information of the holding unit 200. This allows the priority of the holding methods to be adjusted while actually measuring the physical degradation of the holding unit 200, thereby extending the life of the holding unit 200.

[0236] (Sixth embodiment)

[0237] Next, the sixth embodiment will be described. The sixth embodiment differs from the first embodiment in that an algorithm using machine learning is used instead of a rule-based algorithm. Except for the configuration described below, the other configurations are the same as those of the first embodiment.

[0238] In this embodiment, the calculation of the first, second, and third scores in the first embodiment can be performed through machine learning. For example, assuming supervised learning, the aforementioned evaluation function can also be used as the evaluation value during learning. The learning algorithm is not limited to supervised learning and can be modified according to the type of learning, such as unsupervised learning and reinforcement learning.

[0239] Regarding the switching of the holding method, the control device 12 constructs a learning network based on information obtained from the sensor 11, information obtained by the first score, information obtained by the second score, and information required for the number of switching of the holding method, thereby being able to learn to evaluate the success rate of the holding action, the number of switching of the holding method, and the frequency of use of the holding method, and adjust the balance between them. For example, the control device 12 trained by machine learning can receive as input the information obtained in the information acquisition process (step S1) performed by the sensor 11, and output one or more holding strategies suitable for increasing the success rate of the holding action, suppressing the switching of the holding method, and suppressing the deviation in the frequency of use of the holding method, including scores for each holding object and each holding method, and the order of holding and switching.

[0240] When machine learning is used, in the evaluation function of the first score of Formula 2 and Formula 3, let w p1 =w p2 =w p3 =w p4 =w p5 =0,w s1 =w s2 =w s3 =w s4 =w s5 =0.

[0241] Furthermore, regarding configuration change operations, the control device 12 can construct a learning network based on information acquired from the sensor 11, information obtained through the third score, and the success or failure of maintenance, thereby performing learning to optimize the determination of whether to execute a configuration change and improve the effectiveness of configuration change operations. For example, the control device 12 trained by machine learning can receive as input the information acquired in the information acquisition process (step S1) performed by the sensor 11 and output a third score, a configuration change strategy, etc. suitable for improving the determination of whether to execute a configuration change and the appropriateness of selecting an effective configuration change operation, thereby improving system throughput.

[0242] In addition, the learning network related to the switching of the retention method and the learning network related to the configuration change operation may be integrated into one learning network.

[0243] According to the configuration of the sixth embodiment as described above, the control device 12 can efficiently output a more appropriate score, retention strategy, and configuration change strategy by repeatedly performing machine learning.

[0244] In each of the above embodiments, the control device 12 compares the second score T H，H0 (I) and the second threshold Th2 are used to determine the necessity of switching and then determine the order of the holding action and the switching action. However, the necessity of switching may not be determined and the order of the holding action and the switching action may be determined in such a way as to minimize the number of switching. For example, the score calculation unit 304 of the control device 12 may be configured to calculate the first score S indicating the simple ease of holding without considering the currently selected holding method. H (I) Correct only the first score S based on the currently selected holding method H0 H0 The value of (I) (for example, set to n times (n>1)), the first score S based on other holding methods H (I) Generate the second score T as the original value H，H0 (I), based on the second score T H，H0 (I) A holding strategy including the order of holding actions and switching actions is determined in such a way as to reduce the number of switching times as much as possible. In this case, the control device 12 does not need to judge the necessity of switching the holding method. H The second score T of the correction method (I) H，H0 The generation method of (I) is not limited to the above example, and any method can be used as long as some bias is applied to give priority to the currently selected holding method.

[0245] In the above embodiments, the transport device 10 includes a first transport device 10A that performs "gripping" and a second transport device 10B that performs "adsorption." However, the structure of the transport device 10 is not limited to this. As long as the structure can select one or more of multiple holding methods to hold an object, the transport device 10 may have a structure having one or more holding units that perform any holding method. For example, the transport device 10 may include other transport devices in addition to the first transport device 10A and the second transport device 10B. The transport device 10 may also include a hybrid robot capable of both "gripping" and "adsorption" in place of the first and second transport devices 10A, 10B. Furthermore, at least one of the first and second transport devices 10A, 10B may be a hybrid robot capable of both "gripping" and "adsorption." Such a hybrid robot may, for example, have a structure that switches between the gripping and adsorption units by rotating a 180° rotating unit, a volver-type structure like that of an optical microscope, or other structures. The transport device 10 may include two types of suction transport devices having different sizes, shapes, characteristics, etc. Alternatively, the transport device 10 may include a transport device that holds the object O using a holding method other than clamping and suction.

[0246] According to at least one embodiment described above, the effectiveness of the arrangement change operation can be improved, thereby shortening the operation time of the transport device and improving the throughput of the system.

[0247] While several embodiments of the present invention have been described above, these embodiments are provided as examples and are not intended to limit the scope of the invention. These embodiments may be implemented in various other ways, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments or modifications thereof are intended to be within the scope and spirit of the invention and are encompassed by the invention as set forth in the claims and their equivalents.

Claims

1. A transport system capable of transporting a plurality of objects, wherein: The handling system includes: movable arm; a holding portion mounted on the movable arm and capable of holding the object, the holding portion including a plurality of holding mechanisms enabling selection of one or more of a plurality of holding methods; a sensor capable of detecting the object; and a control unit that controls the movable arm and the holding unit; The control unit determines whether to change the arrangement of the object before holding the object based on the information obtained from the sensor. When it is determined that a configuration change of the object is to be performed, a holding method and a configuration change operation are determined based on a holding obstacle that makes holding of the object difficult, the effectiveness of the determined configuration change operation when the determined holding method is used is evaluated for each object, and a configuration change strategy is determined based on the evaluation result, the configuration change strategy including: (1) a configuration change operation; (2) a holding method of a holding mechanism that is effective for the configuration change operation; and (3) an object that is the subject of the configuration change operation.

2. The transport system according to claim 1, wherein: The control unit calculates a score of the arrangement change operation for each of the objects in the evaluation, and selects an object to be the arrangement change target based on the score.

3. The transport system according to claim 2, wherein: The score of the configuration change operation calculated for each object is calculated based on at least one of the ease of the configuration change operation of the object, the difficulty of maintaining the object, and the number of objects predicted to be configured differently following the configuration change operation of the object.

4. The transport system according to any one of claims 1 to 3, wherein: The control unit estimates a retention hindering factor that hinders retention of the object based on information acquired from the sensor, and determines a placement change operation based on the estimated retention hindering factor and a result of the evaluation.

5. The transport system according to any one of claims 1 to 3, wherein: The control unit determines a degree of density of the objects based on at least one of an arrangement and a shape of the objects, and determines an arrangement change operation so that the degree of density of the objects becomes smaller.

6. The transport system according to any one of claims 1 to 3, wherein: The holding portion is configured to hold the object by at least one of clamping and adsorption as the holding method. The control unit determines the configuration change operation so that the distance between the objects increases when the holding unit holds the object by clamping, and determines the configuration change operation so that the area capable of adsorbing the object increases when the holding unit holds the object by adsorption.

7. The transport system according to any one of claims 1 to 3, wherein: The holding method currently selected by the holding unit is used as the holding method of the holding unit effective for the configuration change operation.

8. The transport system according to any one of claims 1 to 3, wherein: The control unit calculates a holding ease score for each of the objects based on the information acquired from the sensor, and determines whether to change the arrangement of the objects based on the holding ease score.

9. The transport system according to claim 8, wherein: The control unit determines to change the arrangement of the objects when the objects having the holding ease score being equal to or greater than a predetermined threshold value do not exist.

10. The transport system according to any one of claims 1 to 3, wherein: The control unit selects one or more configuration change operations from a plurality of configuration change operations based on at least one of the information about the object and the information acquired from the sensor, and evaluates the effectiveness of the configuration change operation for each of the objects with respect to the selected configuration change operation.

11. The transport system according to claim 10, wherein: The control unit selects one or more configuration change operations from a plurality of configuration change operations based on one or more information selected from the group consisting of the softness of the object, the surface area of ​​the portion of the object predicted to be exposed after the configuration change operation, the aspect ratio of the object, the height difference between the object and its surroundings, and the size of the space in the surrounding area of ​​the object.

12. The transport system according to any one of claims 1 to 3, wherein: The control unit calculates a position at which a determined arrangement change operation is to be performed and a posture of the movable arm for performing the determined arrangement change operation.

13. The transport system according to any one of claims 1 to 3, wherein: The control unit calculates a score for determining a priority of a holding operation for each of the objects based on information acquired from the sensor, and corrects the score for determining the priority based on information of the object whose holding operation failed when it is determined that the holding unit has failed.

14. A conveying system capable of conveying a plurality of objects, wherein: The delivery system includes: movable arm; a holding portion mounted on the movable arm, capable of holding the object, comprising a plurality of holding mechanisms enabling selection of one or more holding methods; a sensor capable of detecting the object; and a control unit that controls the movable arm and the holding unit; The control unit determines whether to change the arrangement of the object before holding the object based on the information obtained from the sensor. When the control unit determines that a configuration change of the object is to be performed, the control unit determines a holding method and a configuration change operation based on a holding obstacle that makes holding the object difficult, evaluates the effectiveness of the determined configuration change operation when the determined holding method is used for each object, and determines a configuration change strategy based on the evaluation result, the configuration change strategy including: (1) a configuration change operation; (2) a holding method of a holding mechanism that is effective for the configuration change operation; and (3) an object that is the subject of the configuration change operation. When determining that the arrangement of the object is not to be changed, the control unit calculates a position for holding the object selected as a holding target and a posture of the movable arm, and moves the object from a first position to a second position.

15. A control device for controlling a transport system capable of transporting a plurality of objects, wherein: The transport system comprises: movable arm; a holding portion mounted on the movable arm, capable of holding the object, and including a plurality of holding mechanisms enabling selection of one or more holding methods; and a sensor capable of detecting the object; The control device determines whether to change the arrangement of the object before holding the object based on the information obtained from the sensor. When it is determined that a configuration change of the object is to be performed, a holding method and a configuration change operation are determined based on a holding obstacle that makes holding of the object difficult, the effectiveness of the determined configuration change operation when the determined holding method is used is evaluated for each object, and a configuration change strategy is determined based on the evaluation result, the configuration change strategy including: (1) the configuration change operation; (2) a holding method of a holding mechanism that is effective for the configuration change operation; and (3) an object that is the object of the configuration change operation.

16. A storage medium, which is a non-volatile computer-readable storage medium storing a program for controlling a transport system capable of transporting a plurality of objects, wherein: The transport system comprises: movable arm; a holding portion mounted on the movable arm, capable of holding the object, comprising a plurality of holding mechanisms enabling selection of one or more holding methods; a sensor capable of detecting the object; and a control unit that controls the movable arm and the holding unit; The program causes the control unit to execute the following steps: determining, based on information acquired from the sensor, whether to change the configuration of the object before holding the object; and When the control unit determines that a configuration change of the object is to be performed, a holding method and a configuration change operation are determined based on a holding obstacle that makes holding of the object difficult, and the effectiveness of the determined configuration change operation when the determined holding method is used is evaluated for each object. Based on the result of the evaluation, a configuration change strategy is determined, and the configuration change strategy includes: (1) a configuration change operation; (2) a holding method of a holding mechanism that is effective for the configuration change operation; and (3) an object that is the subject of the configuration change operation.

17. A transport method using a transport system capable of transporting a plurality of objects, wherein: The transport system comprises: movable arm; a holding portion mounted on the movable arm, capable of holding the object, comprising a plurality of holding mechanisms enabling selection of one or more holding methods; a sensor capable of detecting the object; and a control unit that controls the movable arm and the holding unit, The control unit determines whether to change the arrangement of the object before holding the object based on the information obtained from the sensor. When the control unit determines that a configuration change of the object is to be performed, a holding method and a configuration change operation are determined based on a holding obstacle that makes holding of the object difficult, and the effectiveness of the determined configuration change operation when the determined holding method is used is evaluated for each object. Based on the result of the evaluation, a configuration change strategy is determined, and the configuration change strategy includes: (1) a configuration change operation; (2) a holding method of a holding mechanism that is effective for the configuration change operation; and (3) an object that is the subject of the configuration change operation.

Images