Control device and control method for robot system

Abstract

To provide a control device, a control method, and the like for a robot system that realizes a high degree of cooperation between units.SOLUTION: There is provided a device for controlling a robot system including an operation robot and a transfer robot for storing or taking out an operation target in a rack, including: a data acquisition unit that acquires first data including information about the operation target and the rack before storing or taking out the operation target in the rack; a data storage part; and a robot control unit that creates or acquires a control sequence for selecting the rack and transfers it to the access position before storing or taking out the operation target in the rack, as well as for storing or taking out the operation target in the rack, instructs the transfer robot to execute a task to transfer the rack to the rack access position, and instructs the operation robot to execute a task to store or take out the operation target in the rack.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure generally relates to a robot system, and more particularly to a control device, a control method, a logistics system, a program, and a recording medium for a robot system that manipulates an object such as an article. [Background technology]

[0002] Many robots (e.g., machines configured to automatically / independently perform physical operations) are now widely used in many fields due to their ever-improving performance and decreasing cost. For example, robots can be used to perform various tasks, such as manipulating and moving objects, in manufacturing, assembly, packaging, transporting, conveying, shipping, etc. In performing tasks, robots can replicate human actions, thereby replacing or reducing dangerous or repetitive tasks performed by humans.

[0003] As an example of a system using such a robot (robot system), Patent Document 1 proposes an automated logistics system that includes a transport container storage mechanism that temporarily stores transport containers, and an automatic item sending mechanism that automatically collects items in transport containers into shipping containers based on shipping information, in order to automate and reduce the labor required for the process from receiving to shipping items. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-167950 Summary of the Invention [Problem to be solved by the invention]

[0005] However, despite technological advances, robots often lack the sophistication necessary to replicate human intervention to perform larger and / or more complex tasks. As a result, automation and functionality in robotic systems remain insufficient, and there are many tasks that are difficult to replace human intervention, and tasks that are physically possible to replace with robots but have not been automated due to efficiency concerns. Therefore, there remains a need for technological improvements to manage various actions and / or interactions between robots and further promote automation and functionality in robotic systems. Therefore, an object of the present disclosure is to provide, for example, a control device and control method for a robotic system that achieves advanced cooperation between units, including robots. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention employs the following configuration.

[0007] [1] In other words, the control device according to the present disclosure is a device for controlling a robot system including an operating robot that operates an operating object, such as a fetching robot, a de-bunching robot, a piece-picking robot, or the like, using a robot arm and an end effector to store the operating object in a rack and / or remove the operating object from the rack, and a transport robot that transports the rack. The control device includes: (1) a data acquisition unit that acquires first data including information about the operation target and information about the rack before storing the operation target in the rack and / or removing the operation target from the rack; (2) a data storage unit that stores the first data; and (3) a robot control unit that selects the rack and transports it to an access position based on the first data before storing the operation target in the rack and / or removing the operation target from the rack, and creates or acquires a control sequence for storing the operation target in the rack and / or removing the operation target from the rack, and commands the transport robot to execute a task for transporting the rack to the access position based on the control sequence, and commands the operation robot to execute a task for storing the operation target in the rack and / or removing it from the rack.

[0008] Here, the term "operation target" refers to an object operated by an operation robot provided in a robot system, and includes, for example, one or more items, and containers such as bottles, containers, and boxes on which the items are placed or housed. The containers may be packaged or unpackaged, and may have a portion (e.g., the top) of the container open (so-called "top-cut" containers). In other embodiments and examples, the term "operation target" may also refer to a shelf, pallet, conveyor, or other temporary storage area. Furthermore, a "rack" transported by a transport robot provided in a robot system refers to a device having one or more shelves on which at least one operation target can be held. Furthermore, "information about the operation target" and "information about the rack" refer to information associated with the identification information of the operation target and information associated with the identification information of the rack, respectively. Furthermore, the "access position" of the "rack" refers to a position at which the operation robot can access the rack. Furthermore, "control sequence" refers to the order of operations that is set in advance when controlling one or more units in a robot system to execute individual tasks (both the operation robot and the transport robot are types of "units"). Furthermore, "before" refers to a point in time or before that point in time, and "after" refers to a point in time or after that point in time.

[0009] According to this configuration, before storing the operation object in the rack and / or before removing the operation object from the rack, information about the operation object and the rack is grasped, and a task is executed based on a control sequence created based on first data including that information. Therefore, the operation of storing the operation object in the rack and / or the operation of removing the operation object from the rack can be carried out efficiently and smoothly. Furthermore, at that time, cooperation between units in the robot system, particularly high-level cooperation between the operation robot and the transport robot, can be realized, thereby improving the functionality of the robot system.

[0010] [2] In the above configuration, the data acquisition unit may acquire second data including information about the operation object and information about the rack after storing the operation object in the rack and / or removing the operation object from the rack, and the data storage unit may be configured to store the second data.

[0011] With this configuration, after the operation target is stored in the rack and / or after the operation target is removed from the rack, information about the operation target and the rack can be grasped, thereby making it possible to reliably grasp and track the state or situation of the rack after the operation target is stored in and / or after the operation target is removed.

[0012] [3] In the above configuration, the robot control unit may be configured to create or acquire the control sequence based on the second data so that the arrangement of the operation objects on the rack is consolidated.

[0013] According to this configuration, after an operation target is stored in a rack and / or removed from the rack, rack availability information can be grasped from the second data related to the operation target and the rack. Therefore, as a next step, it is possible to efficiently select a rack when storing (filling) an operation target in a rack and / or removing an operation target from a rack. As a result, it is possible to improve the storage efficiency of operation targets in a rack and prevent fragmentation of wasted space on the rack (defragmentation). Furthermore, it is possible to suppress the occurrence of such wasted space and further facilitate centralized management of location information of operation targets and racks.

[0014] [4] In the above configuration, the robot control unit may be configured to instruct the operation robot to perform at least a part of the operations in the control sequence before the rack reaches the access position.

[0015] With this configuration, while the transport robot is transporting the rack to the access position (i.e., before the rack arrives at the access position), the operation robot can "preempt" the operation of storing or retrieving the operation target, which results in more advanced cooperation that can further shorten the operation time.

[0016] [5] In the above configuration, the data acquisition unit may acquire relative position information between the transport robot and the rack, and the robot control unit may be configured to correct the position information of the rack at the access position based on the relative position information, and to create or acquire the control sequence based on the corrected position information.

[0017] With this configuration, when the operation robot accesses the rack, it is possible to correct the positional deviation between the rack and the transport robot holding the rack, regardless of whether the rack or the transport robot is used as the position reference for operation. This allows the operation robot to more accurately perform both the operation of storing the operation target in the rack and the operation of removing the operation target from the rack, further improving work accuracy and work efficiency.

[0018] [6] In the above configuration, the rack is configured so that the object to be operated is stored and / or removed from the side, and a fall prevention device for the object to be operated is provided at the end of the rack, the operating robot grasps the object to be operated from the side (for example, a devanning gripper or the like can be used. Devanning is, for example, the work of unloading from a container), and the robot control unit may create or acquire the control sequence so that the object to be operated moves beyond the fall prevention device. Note that the "end of the rack" refers to the entire circumference or part of the entire circumference near the periphery of the top surface of the shelf of the rack, or refers to a position where at least a part of the object to be operated and at least a part of the fall prevention device interfere with each other when the object to be operated, such as an item, is placed on the shelf.

[0019] According to this configuration, the fall prevention devices provided on the shelves of the rack prevent the operation object placed on the shelves from falling out of the rack. Furthermore, the operation object is grasped by the operation robot from the side, and in that state is stored in and / or removed from the rack from the side. Therefore, even if the rack has multiple shelves and is configured in multiple levels, the operation object can be easily stored in and / or removed from the rack. Furthermore, since the operation robot moves the operation object so as to avoid (go over) the fall prevention devices, the operation object can be stored in and removed from the rack smoothly and reliably.

[0020] [7] In the above configuration, the data storage unit may be configured to store the position information (placement information) of the multiple operation objects in the rack on a per-operation-object basis or per-shelf-board basis in the rack, and as two-dimensional information or three-dimensional information.

[0021] According to this configuration, the position of an operation target placed on a shelf of a rack can be grasped for each operation target or for each shelf (which can also be called a layer). In addition, the arrangement of multiple operation targets can be managed in a unified manner as planar or three-dimensional information across the same shelf or different shelves. Therefore, for example, by identifying one operation target, it is possible to collectively identify operation targets stored on the same shelf of the rack, or operation targets stored in the rack. As a result, for example, it is possible to reduce the number of work steps required for inventory management and inventory taking, thereby improving work efficiency.

[0022] [8] In the above configuration, the robot system may further include a sensor that captures an image of the rack in which the object to be operated is stored, and the robot control unit may be configured to create or acquire the control sequence for storing the object to be operated in the rack so that the sensor can measure the identification information (representing an identification code or identification tag) of the object to be operated.

[0023] According to this configuration, even when the operation target is stored in a rack, the identification information of the operation target can be measured and confirmed by a sensor. In this case, if the identification information of the operation target is attached at least to the side, the identification information can be exposed so that it can be seen from the outside of the shelf. Therefore, the identification information (representing the identification code or identification tag) attached to the operation target can be easily recognized by, for example, capturing an image with a sensor such as a camera. Therefore, even when the operation target is stored in a rack, it can be easily identified and specified. In this case, if the position information of multiple operation targets on the rack is stored as two-dimensional or three-dimensional information as described above in [7], by identifying one operation target, it is possible to more easily identify all operation targets stored in the rack at once, by shelf or rack. As a result, the efficiency of inventory management and inventory taking can be further improved.

[0024] [9] In this case, more specifically, the robot system may further include a sensor that captures an image of the rack in which the object to be operated is stored, and the sensor may be configured to capture an image of the rack when the transport robot holding the rack is stationary or when the transport robot holding the rack is moving.

[0025]

[10] In the above configuration, the robot control unit may be configured to determine that the storing of the operation object into the rack and / or the removal of the operation object from the rack is completed before the storing of the operation object into the rack and / or the removal of the operation object from the rack is completed.

[0026] With this configuration, it is possible to determine the completion of the storing and removing of the operation target into the rack and / or before the operation target is completely removed from the rack, which allows the operation of the task to be executed as the next step to be started smoothly and in a timely manner, preventing work delays and enabling more advanced cooperation between units.

[0027]

[11] In the above configuration, the data acquisition unit may be configured to acquire actual measured values ​​or estimated values ​​of information about the operation object and / or information about the rack. This allows the position of the operation object inside / outside the rack, the position of the rack, and the position of the identification information attached to the operation object to be confirmed by actual measurement using a sensor such as a camera, or can be estimated based on master data without actual measurement.

[0028]

[12] Furthermore, a logistics system according to the present disclosure includes a robot system including the control device having the above-described configuration, the operation robot, and the transport robot.

[0029]

[13] In the above configuration, the control device may be configured to identify the robot system and an area associated with the robot system, and calculate tasks (including unit tasks and tasks formed by combining multiple unit tasks) based on the control sequence, the tasks including tasks related to the transportation of the rack by the robot system and the operation of the operation object, as well as multiple tasks executed across adjacent and / or overlapping areas by the robot system.

[0030]

[14] Furthermore, the program according to the present disclosure is configured as a program for causing a computer to function as a control device for each of the above configurations.

[0031]

[15] Furthermore, the recording medium according to the present disclosure is configured as a non-transitory computer-readable recording medium on which the above program is recorded.

[0032]

[16] Furthermore, the control method according to the present disclosure is a control method for a robot system including an operation robot that operates an operation object and a transport robot that transports the rack in order to store the operation object in a rack and / or remove the operation object from the rack, and is executed using a control device having a data acquisition unit, a data storage unit, and a robot control unit. Then, (1) the data acquisition unit acquires first data including information about the operation target and information about the rack before storing the operation target in the rack and / or removing the operation target from the rack, (2) the data storage unit stores the first data, and (3) the robot control unit selects the rack and transports it to an access position based on the first data before storing the operation target in the rack and / or removing the operation target from the rack, and creates or acquires a control sequence for storing the operation target in the rack and / or removing the operation target from the rack, and commands the transport robot to execute a task for transporting the rack to the access position based on the control sequence, and commands the operation robot to execute a task for storing the operation target in the rack and / or removing the operation target from the rack. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a schematic flow diagram illustrating an exemplary task sequence in which a robotic system may operate according to an embodiment of the present disclosure. [Figure 2] 1A and 1B are perspective views schematically illustrating the exterior of an exemplary transport robot according to one or more embodiments of the present disclosure. [Figure 3] FIG. 1 is a perspective view schematically illustrating the exterior of an exemplary manipulation robot in accordance with one or more embodiments of the present disclosure. [Figure 4] 1A and 1B are perspective views schematically illustrating the appearance of an exemplary manipulation robot according to one or more embodiments of the present disclosure. [Figure 5]FIG. 1 is a block diagram illustrating an example of a hardware configuration and a functional configuration of a robot system according to an embodiment of the present disclosure. [Figure 6] 1 is a front view (partial functional configuration diagram) showing an example of a flow of work executed by a robot system according to an embodiment of the present disclosure and an example of a functional configuration of a control device provided in the robot system. FIG. [Figure 7] FIG. 10 is a flowchart illustrating an example of a procedure for operating a robot system according to an embodiment of the present disclosure to perform a first task. [Figure 8] FIG. 10 is a flow diagram illustrating an example of a procedure for operating the robot system according to the embodiment of the present disclosure to perform a second task. [Figure 9A] FIG. 1 is a schematic plan view illustrating an exemplary environment in which a robotic system according to an embodiment of the present disclosure may operate. [Figure 9B] 9B is a schematic plan view of a portion of an exemplary environment in which a robotic system according to an embodiment of the present disclosure may operate, showing the classification area in FIG. 9A and its surroundings. FIG. [Figure 10A] FIG. 10 is a schematic plan view illustrating another exemplary environment in which a robotic system according to an embodiment of the present disclosure may operate. [Figure 10B] 10B is a schematic plan view showing another exemplary environment in which a robotic system according to an embodiment of the present disclosure can operate, illustrating the classification area in FIG. 10A and its surroundings. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0034] According to the present disclosure, there are provided a robot system in which multiple units (e.g., various robots, various devices, and control devices provided integrally or separately therefrom) are highly integrated, a control device for the same, a logistics system including the same, and methods therefor, etc. That is, a robot system according to an embodiment of the present disclosure is, for example, an integrated system that can autonomously execute one or more tasks.

[0035] Here, "task" may include, for example, a task for a robot or the like to access an object to be operated, a task for moving, storing, safekeeping, retrieving, organizing, etc., the object to be operated from one location to another, a task for classifying the object to be operated into multiple groups, a task for photographing, observing, and managing the object to be operated, etc.

[0036] Furthermore, a "task" may include a combination of multiple tasks executed in the warehousing of operation objects in a logistics system (receiving, transporting, palletizing (stowing), depalletizing (unloading), storing, etc.), the replenishment of operation objects (opening, filling, replenishing, etc.), and the elimination of operation objects (removing, picking up, packing, stowing, palletizing, etc.). Furthermore, a "task" may include, for example, grasping or lifting an operation object at a specified position, moving it along a specified route, and releasing, lowering, or placing it at a specified position in order to rearrange, change, replace, etc. the operation object.

[0037] Furthermore, the robot system according to the present disclosure can access operation targets via multiple units, such as an operation robot and a transport robot, thereby automating tasks such as the receiving process, replenishment process, and retrieval process of the operation targets. Furthermore, the robot system according to the present disclosure can appropriately classify (group) operation targets, move or place them at a destination or designated location, access the operation targets, and rearrange or reclassify them at that location or to another location, rearranging or replacing them according to their characteristics. In this case, the robot system can read one or more identification information (e.g., a barcode or a Quick Response (QR) Code (registered trademark)) attached to one or more specific locations or surfaces of the operation targets and compare it with master data as needed to identify and / or specify the operation targets and obtain information associated with the operation targets.

[0038] Furthermore, the robot system according to the present disclosure may include a sensor, such as an image sensor, for identifying the position and state (e.g., orientation, etc.) of the object and / or the surrounding environment of the object. The image sensor can acquire images of the work positions (e.g., pick-up position, drop position, position along the path, etc.) of the object in the task performed by each unit of the robot system, as well as images of the object at each position. The robot system according to the present disclosure can also process images of the object in a predetermined order (e.g., from the top to the bottom of the object, from the outer edge, from the inside, etc.). In this case, the state and situation of the object can be appropriately determined by identifying and classifying the outer shape and environment of the object based on, for example, the color, brightness, and changes in the values ​​of adjacent pixels in the image of the object.

[0039] The robot system according to the present disclosure can acquire and execute control sequences for performing tasks such as accessing and manipulating an object, and transporting, moving, placing, and storing the object. Such control sequences may include combinations of basic control sequences for driving the operating mechanisms of each unit. The robot system can create or acquire control sequences for performing various tasks by, for example, motion planning, machine learning such as deep learning, and the like.

[0040] For example, conventional robotic systems used in typical logistics systems often lack the ability to fully interact with multiple units when performing tasks related to the storage, replenishment, and delivery of controlled objects. Operator assistance is often required between successively executing different tasks. Furthermore, while conventional robotic systems can access controlled objects in response to orders, operators often need to categorize and sequence the ordered items. Furthermore, conventional robotic systems often struggle to autonomously change the units that operate controlled objects or the operating procedures of controlled objects. In particular, when a system includes units that require tasks that require operator intervention or assistance, autonomously or automatically changing the control sequence is extremely difficult.

[0041] Furthermore, in conventional robot systems, the basic control sequence involves performing tasks such as grasping an object at a certain position, moving it to another position in that state, and then releasing it. However, such basic operations alone have not been able to sufficiently improve the storage efficiency of the object or the work efficiency of each unit. In particular, in order to improve the storage efficiency of the object, there have been cases where operator intervention or assistance (adjustments, redoing, supplementation, system shutdown, etc.) has been required.

[0042] In contrast, the robot system according to the present disclosure in various aspects exemplified below, unlike conventional systems, can coordinate and control the interactions between separate units (e.g., an operation robot and a transport robot) to execute tasks, and can facilitate cooperation between the different units. This reduces the intervention and assistance of an operator that was previously required, and can improve the storage efficiency, work efficiency, and economy of the objects to be operated.

[0043] Furthermore, the robot system according to the present disclosure can reliably identify the operating area, operating path, position and status of the target, or a combination thereof, for each unit, and smoothly execute tasks performed by different units across the board. Furthermore, in this process, the storage efficiency of the target can be optimized based on the shape information, identification information, position information, etc. of the target, thereby further improving space utilization efficiency. To this end, the robot system according to the present disclosure can create or obtain a suitable control sequence that enables each unit to perform a highly coordinated operation based on one or more algorithms that sequence the tasks of different units, one or more protocols that control the interactions between units, and information on the status of the target.

[0044] Hereinafter, an embodiment according to an example of the present disclosure will be described with reference to the drawings. However, the embodiment described below is merely an example and is not intended to exclude various modifications or applications of techniques not explicitly described below. In other words, the example of the present disclosure can be implemented with various modifications within the scope of its spirit. In addition, in the description of the drawings below, identical or similar parts are designated by identical or similar reference numerals, and the drawings are schematic and do not necessarily correspond to actual dimensions, ratios, etc. Furthermore, parts with different dimensional relationships or ratios may be included between the drawings. It goes without saying that the embodiments described below are only some of the embodiments of the present disclosure, and do not represent all of the embodiments. Furthermore, other embodiments that can be obtained by those skilled in the art based on the embodiments of the present disclosure without requiring creative acts are all within the scope of protection of the present disclosure.

[0045] Also, in each embodiment, the technology introduced herein may be practiced without those specific details. Furthermore, well-known functions, such as particular functions or routines, will not be described in detail to avoid unnecessarily obscuring the present disclosure. Also, well-known structures or processes often associated with robotic systems and subsystems may not be described in detail for purposes of clarity. References herein to “an embodiment,” “one embodiment,” or the like mean that a particular feature, structure, material, or characteristic described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases herein do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various illustrated embodiments are merely illustrative representations and are not necessarily drawn to scale.

[0046] Furthermore, many embodiments or aspects of the present disclosure include processes, steps, routines, blocks, etc., executed by a programmable computer or controller, and may take the form of computer-executable or controller-executable instructions. Those skilled in the relevant art will appreciate that the techniques of the present disclosure may be implemented in computer or controller systems other than those shown in the embodiments. The techniques described herein may be implemented within a special-purpose computer or data processor that is programmed, configured, or constructed to execute one or more computer-executable instructions described below.

[0047] Thus, the terms "computer" and "controller" as generally used herein include any data processor, including Internet appliances and handheld devices (including palmtop computers, wearable computers, cellular or mobile telephones, multi-processor systems, processor-based or programmable consumer electronics appliances, network computers, minicomputers, etc.) Information manipulated by these computers and controllers may be provided on any suitable display medium, including, for example, a liquid crystal display (LCD).

[0048] The instructions for performing computer- or controller-executable operations may be stored on any suitable computer-readable non-transitory storage medium, including hardware, firmware, or a combination of hardware and firmware, and may be contained in any suitable memory device, including, for example, a flash drive and / or other suitable medium.

[0049] The terms "coupled," "connected," and similar terms may be used to describe a structural relationship between components. However, these terms are not intended to be synonyms for each other. Specifically, in particular embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise clear from the context, the term "coupled" may be used to indicate that two or more elements are in direct or indirect contact with each other (with other intervening elements therebetween), or that two or more elements cooperate or interact with each other (e.g., in a causal relationship, such as via signal transmission / reception or function calls), or both.

[0050] [Application example] 1 is a schematic flow diagram showing an exemplary work sequence by a logistics system including a robot system according to an embodiment of the present disclosure. The work sequence includes, for example, an in-stock process P10 in which operation objects (in this embodiment, for example, containers such as carton boxes containing items) are received at a logistics center or the like, a replenishment process P20 in which operation objects placed on, for example, pallets are restocked on, for example, racks and stored, and an out-stock process P30 in which ordered items are picked up from the operation objects restocked and stored on the racks and shipped out.

[0051] In the warehousing process P10, first in step S101, the operation object brought in by vehicle or the like is received and unloaded using, for example, a devanning robot. In step S102, for example, a sorting system is used to transport the operation object by conveyor or the like to a waiting position for a pallet for reloading. Next, in step S103, the transported operation object is stacked on an appropriate pallet using, for example, a palletizing robot.

[0052] In the replenishment process P20, in step S104, the pallet on which the operation object is mounted is transported to a position for unpacking using, for example, a pallet AGV (an automated guided vehicle configured to mount or lift and move pallets), and the operation object is then unloaded from the pallet using, for example, a depalletizing robot. In step S105, the operation object is unpacked by cutting (top-cutting) the top surface of the operation object to open it using, for example, a top-cutting machine, and then transported to a standby position on a storage rack using a conveyor or the like. Next, in step S106, the operation object is stored in an appropriate position on the rack using, for example, a fetching robot, and then transported to a storage position on the rack using, for example, a rack AGV (an automated guided vehicle configured to mount or lift and move racks), and then stored in an appropriate state.

[0053] In the outgoing process P30, when an order for an item is placed, in step S107, a rack containing the operation object containing the ordered item is transported to a pick-up position, for example, using a rack AGV, and the desired operation object is picked up from the rack by, for example, a fetching robot. In step S108, the ordered item is picked up from the operation object by, for example, a piece-picking robot, and transferred to a shipping container or the like. Then, in step S109, the shipping container or the like is packed using, for example, a packing machine, and in step S110, the packed shipping container is loaded onto, for example, an appropriate cart or vehicle, and is taken out of the warehouse (shipped).

[0054] As such, the robot system according to an embodiment of the present disclosure may include a de-vanning robot, a palletizing robot, a fetching robot, a piece-picking robot, a packing machine, etc. as operation robots that are transfer / sorting units configured to perform operations to move operation targets between different locations. The robot system according to an embodiment of the present disclosure may also include a sorting system, a pallet AGV, a rack AGV, etc. as transport robots that are transportation units.

[0055] 2A and 2B are perspective views schematically illustrating the exterior of exemplary transport robots 11 and 12 according to one or more embodiments of the present disclosure. The transport robots 11 and 12 may include mobile / wheeled robots such as rack AGVs configured to transport racks between predetermined positions. For example, the transport robots 11 and 12 have external dimensions that enable them to move under and / or between racks. The transport robots 11 and 12 may also include a lifting mechanism configured to lift the racks from the ground (the transport path surface).

[0056] The transport robots 11 and 12 can be guided and steered (navigated) by various mechanisms. For example, the transport robots 11 and 12 can move autonomously to follow a predetermined path, for example, a predetermined path marked as floor markings (e.g., paint, tape, etc.), based on commands from a control device of the robot system according to an embodiment of the present disclosure. The transport robots 11 and 12 can also calculate their current positions via a mapping / positioning mechanism (e.g., a dead reckoning system, a laser-based system, and / or a wireless communication signal-based system) and move along a specified path and route based on the current position. Furthermore, the transport robot 12 may have the function of storing an object to be operated on the rack and / or removing an object from the rack, in addition to transporting the rack itself.

[0057] Note that the robot system according to the embodiment of the present disclosure can transmit the target position of the rack to be transported, the holding position of the rack, the identification information of the rack, a path, an operation plan, or a combination thereof, to the transport robots 11 and 12 via, for example, a control device as a standalone device or part of another unit. Based on the communication command information, the transport robots 11 and 12 can perform tasks such as moving to the holding position of the rack to be transported, lifting the rack, transporting the rack to a specified position, and / or placing the rack at the specified position. Furthermore, the transport robots 11 and 12 can execute or complete the task by, for example, returning the rack to the original holding position or a different storage position.

[0058] 3 and 4 are perspective and side views, respectively, that schematically illustrate the exterior of exemplary manipulation robots 13 and 14 according to one or more embodiments of the present disclosure. The manipulation robots 13 and 14 may include, for example, a robot configured to transfer a manipulation target 16 (see FIG. 4 ) between predetermined positions. For example, the manipulation robots 13 and 14 have structural members such as end effectors 131 and 141, such as grippers, that can grasp the manipulation target by reduced pressure or vacuum suction, and robot arms 132 and 142 to which the end effectors are attached. This allows the manipulation robots 13 and 14 to grasp the manipulation target from the side. The manipulation robots 13 and 14 may be fixed in a predetermined position or may be movable. Additionally, the end effector 141 of the manipulation robot 14 may have a plate-like guide or the like for supporting (holding) the manipulation target 16 from below, such as a so-called de-bunching gripper. This improves the load-bearing capacity and gripping ability when gripping the operation object 16, and as shown in FIG. 4, it becomes easier to handle a relatively large operation object that is transported in a vehicle 40 or the like.

[0059] The manipulation robots 13, 14 can be driven based on, for example, commands from a control device of the robot system according to an embodiment of the present disclosure. The manipulation robots 13, 14 can also calculate the current position of the manipulation target and manipulate the manipulation target along a specified path and route based on the current position. The robot system according to an embodiment of the present disclosure can transmit the target position and pick-up position of the manipulation target, identification information of the manipulation target, a path, a motion plan, or a combination thereof, to the manipulation robots 13, 14 via, for example, a control device as a standalone device or part of a separate unit. Based on the communication command information, the manipulation robots 13, 14 can perform tasks such as moving the end effectors 131, 141 to a grasping position of the manipulation target, grasping and lifting the manipulation target, transporting it to a specified position, and / or placing it at a specified position.

[0060] [Configuration example] 5 is a block diagram showing an example of a hardware configuration and a functional configuration of a robotic system according to an embodiment of the present disclosure. For example, the robotic system 500 may include electronic / electrical devices such as a processor 502, a storage device 504, a communication device 506, an input / output device 508, an actuation device 512, a transport motor 514, a sensor 516, or a combination thereof. The robotic system 500 may include one or more of each of these elements and devices.

[0061] Additionally, each of these elements and devices may be coupled to one another via wired and / or wireless connections. For example, the robotic system 500 may include a bus such as a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus (also known as "Firewire").

[0062] The robotic system 500 may also include, for example, bridges, adapters, controllers, or other signal-related devices to provide wired connections between devices. Wireless connections may be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., wireless fidelity (WIFI)), peer-to-peer or device-to-device communication protocols (e.g., Bluetooth, Near-Field Communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and / or other wireless communication protocols.

[0063] The processor 502 may include a data processor (e.g., a central processing unit (CPU), a special purpose computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored in a storage device 504 (e.g., computer memory). The processor 502 may implement program instructions to control / interact with other devices, thereby causing the robotic system 500 to perform various actions, operations, and / or manipulations in a task.

[0064] Storage device 504 may include a non-transitory computer-readable recording medium having program instructions (e.g., software) stored thereon. Some examples of storage device 504 include volatile memory (e.g., cache and / or random access memory (RAM)), non-volatile memory (e.g., flash memory and / or magnetic disk drives), portable memory drives, cloud storage devices, etc.

[0065] In some embodiments, the storage device 504 may further store and access processing results, predetermined data, predetermined thresholds, predetermined parameters, etc. For example, the storage device 504 may store master data 552 including information data associated with an object that may be manipulated by the robotic system 500.

[0066] The master data 552 may include dimensions, shapes (e.g., computer-generated templates of possible states, computer-generated models for recognizing the object in different states, etc.), color schemes, images, identification information (e.g., barcodes, Quick Response (QR) codes, logos, etc., and their expected locations, etc.), expected weights, or combinations thereof, of objects to be manipulated by the robotic system 500. The master data 552 may also include related information about the objects and their manipulation, such as the location of the center of mass (center of gravity) of each object, expected sensor measurements (e.g., force, torque, pressure, contact metrics, various measurements related thereto, etc.) in response to one or more actions / movements, or combinations thereof.

[0067] Also, for example, the storage device 504 can store object tracking data 554 related to tracking of manipulated objects by the robotic system 500. The object tracking data 554 can include a log of objects that have been scanned or manipulated. The object tracking data 554 can also include image data (e.g., photographs, point clouds, live video feeds, etc.) of the objects at one or more locations (e.g., designated pick-up or drop locations and / or locations on a conveyor belt), and the position and / or state (e.g., orientation) of the objects at the one or more locations.

[0068] The communications device 506 may include circuitry configured to communicate with external or remote devices over a network. For example, the communications device 506 may include a receiver, a transmitter, a conditioner / demodulator (modem), a signal detector, a signal encoder / decoder, a connector port, a network card, etc. The communications device 506 may be configured to transmit, receive, and / or process electrical signals according to one or more communications protocols (e.g., Internet Protocol (IP), wireless communication protocols, etc.). The robotic system 500 may use the communications device 506 to exchange information between units of the robotic system 500 and / or with systems / equipment external to the robotic system 500, for purposes such as data collection, analysis, reporting, troubleshooting, etc.

[0069] The input-output devices 508 may include user interface devices configured to communicate information with and / or receive information from an operator. For example, the input-output devices 508 may include a display 510 and / or other output devices, such as a speaker, haptic circuitry, or tactile feedback device, for communicating information to an operator. The input-output devices 508 may also include control or receiving devices, such as a keyboard, mouse, touchscreen, microphone, user interface (UI) sensors (e.g., a camera for receiving motion commands), wearable input devices, etc. The robotic system 500 may use the input-output devices 508 to interact with an operator in performing tasks, actions, operations, manipulations, or combinations thereof performed by the robotic system 500.

[0070] Suitable units in the robotic system 500 can include structural members articulated for movement (e.g., rotation and / or translation), such as end effectors configured to grip, rotate (spin), weld, etc., and robotic arms configured to manipulate the end effectors (see also, e.g., FIGS. 3 and 4). The robotic system 500 can also include actuation devices 512 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or orient) the structural members at or around the joints, and transport motors 514 configured to transport the units from position to position.

[0071] The robotic system 500 may include sensors 516 for obtaining information used to drive or manipulate the structural members and / or perform a transport task for each unit. The sensors 516 may include various devices configured to detect or measure one or more physical characteristics of the robotic system 500 (e.g., the state, condition, or position of one or more joints or structural members) and / or characteristics of the surrounding environment. Such sensors 516 may include accelerometers, gyroscopes, force sensors, strain gauges, torque sensors, etc., as well as imaging sensors 522, position sensors 524, contact sensors 526, etc.

[0072] The sensors 516 may include one or more imaging sensors 522 configured to detect the surrounding environment, such as visible and / or infrared cameras, two-dimensional and / or three-dimensional imaging cameras (2D vision and / or 3D vision), ranging devices such as lidar or radar, etc. The imaging sensors 522 may generate view data, such as digital images and / or point clouds, that may be used to control units and controls in, for example, automated inspection, robotic guidance, or other robotic applications.

[0073] To manipulate an object, the robot system 500 can acquire and analyze images of a designated area (e.g., an area including a gripping position, pick-up position, drop position, other work position, etc. of the object) to identify each position. For example, the image sensor 522 can include the above-mentioned camera or distance measuring device configured to generate image data and distance data of the designated area. The robot system 500 can determine, for example, a gripping position, pick-up position, drop position, other work position, etc. of the object based on the acquired image and / or distance measurement data. Note that the operation may include scanning the object to log the object during transportation / receiving. In this case, the image sensor 522 can include one or more scanners (e.g., a barcode scanner and / or a QR code scanner (registered trademark)) configured to scan identification information of the object during transportation of the object.

[0074] The sensors 516 may also include position sensors 524, such as position encoders, potentiometers, etc., configured to detect the positions of the structural members (e.g., the robot arm and / or the end effector) and / or joints in each unit of the robot system 500. The robot system 500 may use the position sensors 524 to track the positions and / or states (orientations, etc.) of the structural members and / or joints while performing work in a task.

[0075] Additionally, the sensors 516 may include contact sensors 526, such as pressure sensors, force sensors, strain gauges, piezoresistive / piezoelectric sensors, capacitive sensors, elasto-resistive sensors, other tactile sensors, etc., configured to measure characteristics associated with direct contact between physical structures or surfaces. The contact sensors 526 may measure characteristics corresponding to, for example, gripping of the end effector on the object. Thus, the contact sensors 526 may be configured to detect a physical quantity corresponding to the degree of contact or attachment between the object and the gripper, and may output a quantified measurement (e.g., force, torque, pressure, contact metrics, various measurements thereof, etc.). Note that the contact metrics may include one or more force or torque readings associated with the force applied by the end effector to the object.

[0076] In the above and following descriptions, the robotic system 500 is described as an example in a warehouse or logistics system, but is not limited thereto. The robotic system 500 can be configured to perform various tasks in other environments / for other purposes to perform manufacturing, assembly, packaging, healthcare, and / or other types of automated operations. The robotic system 500 can also include other units, such as manipulators, service robots, and modular robots, which are not shown. The robotic system 500 can also operate, for example, in a warehouse or distribution / transportation hub and include various unloading / loading robots for transferring objects from cage carts or pallets to conveyors or other pallets, sorting systems, unpacking robots for unpacking objects, top-cutting machines, container switching robots for transferring objects from one container to another, packaging robots for packaging objects, packing machines, or combinations thereof.

[0077] [Example of operation] 6 is a front view (partial functional configuration diagram) showing an example of a workflow executed by a robot system according to an embodiment of the present disclosure and an example of a functional configuration of a control device provided in the robot system. The robot system 600 according to this embodiment may be installed in, for example, a warehouse or the like that serves as a receiving, replenishment, storage, and shipping base for a logistics system, and may include a control device 610 as a device that adjusts and controls the operation of each unit in the robot system 600. Furthermore, the robot system 600 can be said to be a system particularly suitable for storing an object to be operated on a rack in the replenishment process P20 shown in FIG. 1 and for removing an object to be operated from the rack in the shipping process P30.

[0078] As described above, the control device 610 is configured as a standalone device or as part of another unit, and can adjust and control the operations of units such as the transport robots 11 and 12 (illustrated as the transport robot 11 in FIG. 6; the same applies below) and the operation robots 13 and 14 (illustrated as the operation robot 13 in FIG. 6; the same applies below) in tasks performed by the units. More specifically, the control device 610 is connected to the transport robot 11, the operation robot 13, and sensors 516 including the imaging sensor 522, and is also connected to a warehouse management system (WMS), other host systems, and / or external systems (not shown), as necessary.

[0079] 5 , the control device 610 is mainly composed of the processor 502, the storage device 504, and the communication device 506. In particular, the processor 502 can function as the data acquisition unit 612 and the robot control unit 616, and the storage device 504 can function as the data storage unit 614. The robot system 600 can be configured to execute a first task for storing the operation object 16 in the rack 15 and / or a second task for removing the operation object 16 from the rack 15.

[0080] [First task: storing the operation target 16 in the rack 15] In the first task, the following tasks A1 to A5 are executed in combination in an appropriate order and at an appropriate timing. A1: The rack 15 stored in the storage area 101 is transported by the transport robot 11 from its storage position to the access position 601 (stop position SL) in the classification area 103. A2: The operation object 16, which is temporarily placed, for example, on a transfer position 602 (transfer position OL) on a conveyor in the classification area 103, is grasped, for example, by suction from the side, by the operation robot 13 (the operation robot 14 may support it from below; the same applies below). A3: The grasped operation target 16 is moved from the transfer position 602 to the access position 601. A4: The operation object 16 is stored (replenished) in a designated vacant position on the rack 15 held by the stationary transport robot 11. A5: The rack 15 to which the operation object 16 has been replenished is returned by the transport robot 11 from the access position 601 to the storage position in the storage area 101.

[0081] 7 is a flow diagram showing an example of a procedure for operating the robot system according to an embodiment of the present disclosure to execute a first task. First, in block 701, the data acquisition unit 612 acquires first data including, for example, information about the operation object 16 and information about each rack 15 before executing task A1, and the data storage unit 614 stores the first data in association with the operation object 16 and the rack 15.

[0082] For example, the data acquisition unit 612 can acquire information about the operation target 16, such as identification information, position information, shape information, weight, and center of mass, by capturing an image of the operation target 16 temporarily placed on the transfer position 602 in the classification area 103 using an imaging sensor 522, such as a 3D vision sensor, and processing the image data, based on the master data 552 as needed. Alternatively, information about the operation target 16 and the rack 15 can be estimated or identified from tracking information about the operation target 16 and the rack 15 in the master data 552. Note that methods for identifying and tracking the operation target 16 can be described, for example, in Japanese Patent Application Nos. 2019-118678 and 2019-080213, U.S. Patent Application No. 16 / 258,120, and the like, filed by the present applicant. Alternatively, the data acquisition unit 612 can acquire information about the operation target 16 from a host system, such as a WMS. In this case, information about the operation target 16 can be acquired in advance before the operation target 16 is placed on the transfer position 602.

[0083] Next, in block 702, the robot control unit 616 determines a rack 15 for storing the operation object 16 based on information about the operation object 16 and information about each rack 15 stored in the storage area 101. In this case, the robot control unit 616 selects suitable candidates for the rack 15 to store the operation object 16, for example, to consolidate the operation objects 16 as much as possible by SKU unit while increasing storage efficiency throughout the warehouse, based on information such as the shape information of the operation object 16, availability information within the rack 15 (information about the space where the operation object 16 is not stored), the SKU (Stock Keeping Unit) of the operation object 16 to be stored, and the SKUs of the operation objects 16 already stored in the rack 15. Furthermore, one or more optimal racks 15 can be determined taking into consideration the transport route to the sorting area 103, etc.

[0084] Next, in block 703, the robot control unit 616 creates or acquires, by machine learning, for example, a control sequence that specifies the transport robots 11 for transporting the determined one or more racks 15 from the storage area 101 to the access position 601 in the sorting area 103, and includes the transport paths of the racks 15 by each transport robot 11. Based on the control sequence, the robot control unit 616 commands the specified transport robots 11 to execute task A1 for transporting the racks 15 to the access position 601.

[0085] Simultaneously with task A1, or before or after task A1, in block 704, the robot control unit 616 creates or acquires, for example, by machine learning, a control sequence for causing the operation robot 13 to grasp the operation object 16, temporarily placed at the transfer position 602, by suction, for example, from the side, and move the operation object 16 from the transfer position 602 to the access position 601, based on information about the operation object 16. Based on the control sequence, the robot control unit 616 commands the operation robot 13 to execute tasks A2 and A3 for grasping the operation object 16 by the operation robot 13 and moving it to the access position 601.

[0086] The timing for executing these tasks A2 and A3 is not particularly limited, and for example, the robot control unit 616 may instruct the operation robot 13 to execute at least a part of the operations in the above control sequence before the rack 15 reaches the access position 601. In this case, the operation robot 13 can complete grasping the operation target 16 and moving it to the access position 601 before the rack 15 reaches the access position 601, and immediately after the rack 15 arrives, task A4 can be executed to store the operation target 16 in the rack 15.

[0087] Next, in block 705, the control device 610, as necessary, uses the imaging sensor 522, such as a 3D vision sensor, to capture images of the operation object 16 and the rack 15 at the access position 601, and processes the image data to confirm the state and status of the operation object 16 and the rack 15. Furthermore, in block 705, the robot control unit 616 creates or acquires, for example, by machine learning, a control sequence that specifies a position on the shelf of the rack 15 for storing the operation object 16 and includes a movement path for the operation object 16 to that position, based on information about the operation object 16 and information about the rack 15 (particularly, the availability of items in the rack 15). Based on the control sequence, the robot control unit 616 commands the operation robot 13 to execute task A4, which is to drop the operation object 16 at a specified, available position on the rack 15 held by the stationary transport robot 11.

[0088] In this case, the robot control unit 616 can set, for example, an arbitrary position of the rack 15 held by the transport robot 11 stopped at the access position 601 as the reference coordinate when the operation robot 13 accesses the rack 15. The resting position of the rack 15 can be calculated based on the access position 601 at which the transport robot 11 holding the rack 15 stops. However, there may be cases where the position of the rack 15 on the transport robot 11 deviates from the standard position. Therefore, the control device 610 calculates the positions of the transport robot 11 and the rack 15 from image data of the rack 15 at the access position 601, and the data acquisition unit 612 acquires relative position information between the transport robot 11 and the rack 15. Then, the robot control unit 616 corrects the position information of the rack 15 at the access position 601 based on the relative position information between them, and preferably creates or acquires a control sequence for task A4 based on the corrected position information of the rack 15 by, for example, machine learning.

[0089] The rack 15 has shelves arranged in multiple stages, and is configured so that the operation target 16 can be stored on or taken out from the shelves from the side, and a fall prevention device 152 having, for example, a convex shape is provided on the end of each shelf of the rack 15. Therefore, the robot control unit 616 can create or acquire, by machine learning or the like, a control sequence including a path for moving the operation target 16 past the fall prevention device 152 so that the moving operation target 16 does not interfere with the fall prevention device 152 when the operation robot 13 is holding the operation target 16 from the side (horizontal direction).

[0090] After the storage (replenishment) of the operation object 16 in the rack 15 is thus completed, in block 706, the robot control unit 616 creates or acquires, by machine learning, for example, a control sequence including a transport path for returning the rack 15, to which the operation object 16 has been replenished, from the access position 601 to a storage position in the storage area 101 by the transport robot 11. Based on the control sequence, the robot control unit 616 commands the transport robot 11 to execute task A5 for transporting and returning the rack 15 to the storage area 101.

[0091] After the first task of storing the operation object 16 in the rack 15 is completed, in block 707, the data acquisition unit 612 acquires second data including information about the rack 15 in which the operation object 16 is stored and information about the storage position of the operation object 16 in the rack 15, and the data storage unit 614 stores the second data in association with the operation object 16 and the rack 15.

[0092] The robot control unit 616 can create or acquire a control sequence based on the second data for the next or subsequent storing (replenishing) operation of the operation objects 16 in the rack 15 so as to consolidate the arrangement of the operation objects 16 on the rack 15, in other words, to store the operation objects 16 densely so as to minimize unnecessary space. In this case, it is also possible to create or acquire a control sequence based on the characteristics of the operation objects 16, such as consolidating operation objects 16 having the same SKU or consolidating and storing operation objects 16 having the same or similar shape and dimensions. The second data related to the storage status of the operation objects 16 on the rack 15 acquired in this manner is updated for each operation object 16 and for each storing operation of the operation object 16, and can be used as appropriate in the second task (removing the operation object 16 from the rack 15) described below.

[0093] Furthermore, based on the second data thus obtained, the data storage unit 614 can store the position (arrangement) information of multiple operation objects 16 in the rack 15 on a per operation object 16 basis or per shelf (layer) basis of the rack 15, and can also store the information as two-dimensional information on the same shelf basis or as three-dimensional information across multiple shelves.

[0094] Furthermore, the operation target 16 may have an identification code or an identification tag on its side as identification information. In this way, even when the operation target 16 is stored in the rack 15, the identification information of the operation target 16 may be visible from outside the rack 15 depending on the arrangement state. Then, by confirming the identification information of the operation target 16 using the image sensor 522 or the like, it is possible to easily identify and specify the operation target 16 even when the operation target 16 remains stored in the rack 15. In this case, if the position information of multiple operation targets 16 in the rack 15 is stored as two-dimensional information or three-dimensional information as described above, it is possible to easily identify the operation targets 16 stored in the rack 15 by identifying one operation target 16, on a shelf-by-shelf basis or on a rack-by-rack basis. In addition, the image sensor 522 can be configured to capture an image of the rack 15 when the transport robot 11 holding the rack 15 is stationary or when the transport robot 11 holding the rack 15 is moving, i.e., when the rack 15 is either stationary or moving.

[0095] Furthermore, the robot control unit 616 may be configured to determine that the storing of the operation object 16 in the rack 15 is complete before the work of storing the operation object 16 in the rack 15 (task A4) is actually completed. For example, by tracking the movement of the operation object 16 based on the operating state of the end effector of the operation robot 13 or by using the image sensor 522, it can be determined that the storing of the operation object 16 is complete when all or most of the operation object 16 has moved onto the shelf of the rack 15, or when part of the operation object 16 has come into contact with the shelf.

[0096] The robot system 600 can perform a first task (storing the operation object 16 into the rack 15) and a second task (removing the operation object 16 from the rack 15) by combining the unit tasks B1 to B5 shown below in an appropriate order and at an appropriate timing.

[0097] [Second task: Removal of operation target 16 from rack 15] In the second task, the following tasks B1 to B5 are executed in combination in an appropriate order and at an appropriate timing. B1: A rack 15 stored in the storage area 101, for example, containing an operation object 16 ordered for shipment, is transported by the transport robot 11 from its storage position to an access position 601 (stop position SL) in the classification area 103. B2: The operation object 16 on the rack 15 held by the transport robot 11, which is stationary at the access position 601, is sucked by the operation robot 13 from, for example, the side direction and grasped. B3: The grasped operation target 16 is moved from the access position 601 to the transfer position 602. B4: The operation target 16 is dropped into the transfer position 602 and temporarily placed there. B5: The rack 15 from which the operation target 16 has been removed is returned by the transport robot 11 from the access position 601 to the storage position in the storage area 101.

[0098] 8 is a flow diagram showing an example of a procedure for operating the robot system according to an embodiment of the present disclosure to execute the second task. First, in block 801, the data acquisition unit 612 acquires first data including, for example, information about the operation object 16 before executing task B1 and information about each rack 15. The data storage unit 614 stores the first data in association with the operation object 16 and the rack 15.

[0099] For example, when a shipping order for the operation object 16 is received, the data acquisition unit 612 can acquire information about the operation object 16, such as identification information, position information, shape information, weight, center of mass, etc., and identification information and storage position information about the rack 15 in which the operation object 16 is stored, based on the second data stored in the data storage unit 614 and, if necessary, the master data 552. Then, in block 802, the robot control unit 616 can select a rack 15 from which to retrieve the operation object 16 based on the information about the operation object 16 and information about each rack 15 stored in the storage area 101, and can determine one or more optimal racks 15 taking into consideration the transport route to the sorting area 103, etc.

[0100] Next, in block 803, the robot control unit 616 creates or acquires, by machine learning, for example, a control sequence that specifies the transport robots 11 for transporting the determined one or more racks 15 from the storage area 101 to the access position 601 in the sorting area 103, and includes the transport paths of the racks 15 by each transport robot 11. Based on the control sequence, the robot control unit 616 commands the specified transport robots 11 to execute task B1 for transporting the racks 15 to the access position 601.

[0101] If the end effector of the manipulation robot 13 is not at the access position 601, simultaneously with task B1, or before or after task B1, the robot control unit 616 commands the manipulation robot 13 to execute a task of moving the end effector of the manipulation robot 13 to the access position 601. In this case, by completing the movement of the manipulation robot 13 before the rack 15 reaches the access position 601, task B2 can be executed to take the manipulation target 16 out of the rack 15 immediately after the rack 15 arrives.

[0102] When the transport robot 11 arrives at the access position 601, in block 804, the control device 610 uses the imaging sensor 522, such as a 3D vision sensor, to capture images of the operation target 16 and the rack 15 at the access position 601, and processes the image data to confirm the state and status of the operation target 16 and the rack 15. Further in block 804, the robot control unit 616 creates or acquires, for example by machine learning, a control sequence for picking up the operation target 16 and moving it from the access position 601 to the transfer position 602, including specifying the position of the operation target 16 on the rack 15 and the movement path of the end effector to that position, based on information about the operation target 16 and information about the rack 15 (particularly the storage status in the rack 15). Based on the control sequence, the robot control unit 616 instructs the operation robot 13 to execute tasks B2 and B3 to grasp the operation object 16 on the rack 15 held by the transport robot 11 stationary at the access position 601 and move it to the transfer position 602.

[0103] At this time, similarly to the first task, the control device 610 calculates the positions of the transport robot 11 and the rack 15 from image data of the rack 15 at the access position 601, and the data acquisition unit 612 can acquire relative position information between the transport robot 11 and the rack 15. Then, the robot control unit 616 corrects the position information of the rack 15 at the access position 601 based on the relative position information between the two, and preferably creates or acquires a control sequence for task B2 by, for example, machine learning, based on the corrected position information of the rack 15. Furthermore, the robot control unit 616 can create or acquire, for example, by machine learning, a control sequence including a path for the moving operation target 16 to move past the fall prevention tool 152 so as not to interfere with the fall prevention tool 152.

[0104] Next, in block 805, the control device 610, as necessary, uses the imaging sensor 522, such as a 3D vision sensor, to capture images of the operation target 16 held at the transfer position 602 and the transfer position 602 (e.g., on a conveyor, a final position, etc.), and processes the image data to confirm the state and situation of the operation target 16 and the transfer position 602. In block 805, the robot control unit 616 further creates or acquires, by machine learning, for example, a control sequence including a designated position to which the operation target 16 is to be transferred and a movement path of the operation target 16 to the designated position. Based on the control sequence, the robot control unit 616 commands the operation robot 13 to execute task B4 for dropping the operation target 16 at the designated position at the transfer position 602. Note that when the operation target 16 is directly transferred to a shipping container or the like prepared at the transfer position 602, the imaging sensor 522 may be used to check the availability of space in the shipping container to improve storage efficiency.

[0105] After the removal of the operation target 16 from the rack 15 is thus completed, in block 806, the robot control unit 616 creates or acquires, by machine learning, for example, a control sequence including a transport path for returning the rack 15 from which the operation target 16 has been removed, by the transport robot 11 from the access position 601 to a storage position in the storage area 101. Based on the control sequence, the robot control unit 616 commands the transport robot 11 to execute task B5 for transporting the rack 15 to the storage area 101.

[0106] After the second task of removing the operation object 16 from the rack 15 is completed, in block 807, the data acquisition unit 612 acquires second data including information about the rack 15 from which the operation object 16 was removed and information about the storage positions of other operation objects 16 stored in the rack 15, and the data storage unit 614 stores the second data in association with the operation object 16 and the rack 15. In this way, the second data about the storage status of the operation object 16 in the rack 15 is updated for each operation object 16 and each time the operation object 16 is removed. Also, similar to the first task, the data storage unit 614 stores position information of the multiple operation objects 16 in the rack 15 for each operation object 16 or for each shelf of the rack 15 based on the second data, and can store the information as two-dimensional information or three-dimensional information.

[0107] Also, similar to the first task, if the operation target 16 has identification information on its side, even when the operation target 16 is stored in the rack 15, depending on the arrangement state, the identification information of the operation target 16 can be confirmed by the image sensor 522, and the operation target 16 can be easily identified by actual measurement even when the operation target 16 remains stored in the rack 15. Furthermore, the image sensor 522 can be configured to capture an image of the rack 15 when the transport robot 11 holding the rack 15 is stationary or when the transport robot 11 holding the rack 15 is moving.

[0108] Furthermore, the robot control unit 616 may be configured to determine that removal of the operation object 16 from the rack 15 is complete before the operation (task B2) of removing the operation object 16 from the rack 15 is actually completed. For example, by tracking the movement of the operation object 16 based on the operating state of the end effector of the operation robot 13 or by using the image sensor 522, it can be determined that removal of the operation object 16 is complete when all or most of the operation object 16 has moved outside the rack 15, or when all or part of the operation object 16 has left the shelf, etc.

[0109] According to the control device 610 and control method of the robot systems 500, 600 configured as described above, before storing the operation object 16 in the rack 15 or before removing the operation object 16 from the rack 15, information about each of the operation object 16 and the rack 15 is grasped, and tasks (first and second tasks) are executed based on a control sequence created or acquired, for example, by machine learning, based on first data including that information. This makes it possible to efficiently and smoothly perform the operation of storing the operation object 16 in the rack 15 or the operation of removing the operation object 16 from the rack 15. Furthermore, at this time, advanced cooperation between units in the robot systems 500, 600 (for example, the transport robot 11 and the operation robot 13) can be realized, thereby enabling the robot systems 500, 600 to have high functionality.

[0110] [Another application example 1] FIG. 9A is a schematic plan view illustrating an exemplary environment in which a robotic system according to an embodiment of the present disclosure can operate. Similar to the robotic system 600, the robotic system 100 can be installed in, for example, a warehouse or the like, which is a receiving, replenishment, storage, and shipping base of a logistics system. The robotic system 100 can be configured to perform one or more tasks and / or a task consisting of a combination of multiple appropriate tasks. These tasks can be suitably performed or executed by the robotic system 100 including one or more units (e.g., various robots, various devices, and control devices integrated with or separately provided therewith). The robotic system 100 can also be applied to the transfer of ordered items from an operation target to a shipping container in the shipping process P30 shown in FIG. 1 .

[0111] As shown in FIG. 9A, the robot system 100 may include one or more transport robots 11, 12 (e.g., rack AGVs and AGVs configured to remove containers from racks and load them onto racks and place them in predetermined positions) as transport units, one or more operation robots 13 (e.g., piece-picking robots) as transfer / sorting units configured to move operation objects between different locations, and their control devices (not shown in FIG. 9A).

[0112] The tasks performed by the robot system 100 can be configured so that each unit performs individual tasks in an appropriate order to achieve a predetermined purpose, or so that each unit can select and perform various different tasks as appropriate. In the robot system 100, each unit can perform one or more tasks to access various different items stored in racks 15 or containers 16 in the storage area 101 and / or various different items classified based on a predetermined storage order, packing order, or item characteristics.

[0113] The transport robot 11 can perform a task of transporting, for example, a rack 15 loaded with containers 16 containing ordered items between the storage area 101 and the transport area 102. On the other hand, the transport robot 12 can remove (pick up) the containers 16 containing ordered items from the transported rack 15. The transport robot 12 can also perform a task of transporting the containers 16 between the transport area 102 and the sorting area 103 (an area for grouping items: the stopping position SL of the containers 16 in FIG. 9A). On the other hand, the operation robot 13 can perform a task of picking up the ordered items from the containers 16 in the sorting area 103 and moving them to a transfer position OL (e.g., a container for packing the ordered items, a box, a conveyor carrying them, a temporary storage area, etc.) and / or another shipping container, etc.

[0114] When performing a task, for example, when a shipping order is placed, when stored items are rearranged, when items are replenished, etc., the robot system 100 can identify different areas in which each unit and / or a group of units operates. For example, the robot system 100 can identify a storage area 101 in which the transport robot 11 operates, a transport area 102 in which the transport robot 12 operates, and a sorting area 103 in which the transport robot 12 and / or the operation robot 13 operate. Note that the areas in which the transport robots 11, 12, and the operation robot 13 operate are not limited to those described above. For example, the transport robot 11 may also operate in the transport area 102, and the transport robot 12 may also operate in the storage area 101, according to an appropriate control sequence.

[0115] The areas in which each unit operates may be adjacent to each other or may partially overlap. For example, the storage area 101 may be adjacent to the transport area 102, and the transport area 102 may partially overlap with the sorting area 103. In this case, the transport robots 11 and 12 can operate in separate areas by accessing racks 15 in the transport area 102, as shown in FIG. 9A . As a result, the robot system 100 can reduce the possibility of different types of units potentially colliding with each other and / or obstructing each other. The operation robot 13 can be fixed in a predetermined position, which makes it easier for the transport robot 12 to enter the sorting area 103 without colliding with other units or causing congestion during movement.

[0116] The robot system 100 can also use appropriate paths to guide and steer transport units such as the transport robots 11 and 12. For example, the robot system 100 can use a first path P1 for steer- ing one or more transport robots 11 and / or a second path P2 for steer- ing one or more transport robots 12. Note that the first path P1 and the second path P2 may be set to be separated by a distance or space so as not to overlap each other, as shown in Fig. 9A, for example, but are not limited to this.

[0117] The first path P1 and the second path P2 may also be provided with floor markings (e.g., paint, tape, etc.) that transport units such as the transport robots 11 and 12 can follow when moving autonomously. This allows the robot system 100 to identify the shape / position of the floor markings and use the identified information when instructing the transport units on route calculation and / or the sequencing of positions (e.g., pick-up positions and / or drop positions for transported objects). Furthermore, the first path P1 and the second path P2 may include a series of links (e.g., paths between racks 15) and nodes (e.g., path intersection positions or designated positions when changing the direction of travel).

[0118] The robot system 100 can also calculate an efficient route for transport units such as the transfer robots 11 and 12 to travel between a pick-up position and a drop position of a transported object without interfering with other units. The transport units can calculate their current position within the route, for example, using a position / navigation system, and move along the specified route and path based on their current position.

[0119] As described above, the robot system 100 can set a task as a series of tasks to be performed by multiple units, and can adjust and control each unit based on the characteristics of each task so as to optimize the execution of the integrated task. For example, adjustments can be made to specify a range of operations that requires multiple units, or to specify a range of operations that spans multiple specified areas.

[0120] For example, to fulfill a shipping order, tasks may be configured and executed to fulfill the order by identifying the storage location of the ordered items in storage area 101, e.g., the rack 15 on which the items are stored or the rack 15 on which the container 16 containing the ordered items is stacked, transporting the rack 15 to the transport area 102, transporting the container 16 from the rack 15 to the sorting area 103, transferring the ordered items from the container 16 to a target location, e.g., a shipping container and / or conveyor, returning the container 16 to the rack 15, returning the rack 15 to storage area 101, or any combination of these tasks.

[0121] In addition, to accommodate the entry and replenishment of goods, available storage locations can be determined, for example, according to demand forecasts, by determining a container 16 capable of storing the goods and a rack 15 capable of storing the container 16 as the storage location for the goods, storing the entered or replenished goods in the container 16, stacking the container 16 containing the entered or replenished goods on the rack 15, transporting the rack 15 to a predetermined corresponding storage location, or setting and executing a task that performs an appropriate combination of these tasks.

[0122] Furthermore, in order to operate each unit, the robot system 100 can generate, for example, instructions (commands) for operating an actuator in each unit, setting conditions, motion plans, positions, identification information, paths, current unit positions, status and progress information, or combinations thereof. The generated information can be communicated between each unit, and the generated information can be shared between the control device and each unit to execute tasks.

[0123] FIG. 9B is a schematic plan view showing a portion of an exemplary environment in which a robot system according to an embodiment of the present disclosure can operate, illustrating the sorting area 103 in FIG. 9A and its surroundings. The sorting area 103 corresponds to an area where the handling robot 13 picks up items from the container 16 and transfers them to a transfer position OL on a conveyor or the like, and / or rearranges the items in the container 16. The sorting area 103 may include, for example, one or more stop positions SL where the transport robot 12 moves to a position near the handling robot 13 and stops. Each stop position SL corresponds to a reference position where each of the one or more transport robots 12 remains stationary while the handling robot 13 performs a task of manipulating an item. Furthermore, each stop position SL includes a position where an end effector (e.g., a gripping tool (e.g., a damper, a de-bunning gripper, etc.) or a hand, etc.) provided at the tip of the robot arm of the handling robot 13 can access the container 16 on the transport robot 12.

[0124] The robotic system 100 may also include an imaging sensor (not shown in FIGS. 9A and 9B) configured to image one or more locations. The imaging sensor may include, for example, a two-dimensional / three-dimensional camera (2D / 3D vision), lidar, radar, etc. configured to image the containers 16 in the sorting area 103 (e.g., on the transport robot 12 at the stop position SL and / or at the transfer position OL). The robotic system 100 may process and use the image data acquired by the imaging sensor for various purposes.

[0125] For example, the robot system 100 may be used to identify an item, determine an item or a combination of items, grasp an item, and determine the position and state (orientation, etc.) of an item. The image data can be processed and used to identify the container 16. Furthermore, the robotic system 100 can use the image data to calculate a status associated with the container 16, such as the number of items stored in the container 16, and whether the container 16 is full or empty. The internal status of the container 16 can be determined by the number of items and / or the height of the items stored in the container 16, etc.

[0126] As a more specific example, if an order is placed containing four different items, the control device of the robot system 100 sends commands based on control sequences corresponding to the respective operations to four different transport robots 12 that pick up four containers 16 containing the ordered items. The four transport robots 12 that pick up and transport the containers 16 corresponding to the four ordered items can stop at one of the stop positions SL in the sorting area 103, as shown in FIG. 9B, for example. When the containers 16 arrive at the sorting area 103 in this way and the transport robots 12 stop at the stop position SL as shown in FIGS. 9A and 9B, each container 16 functions as a container for dispensing or receiving items.

[0127] The robotic system 100 can then operate the handling robot 13 to, for example, pick up the order items from each bin 16 and place them onto a shipping bin or transport conveyor corresponding to the order. Additionally, the robotic system 100 can include one or more units (e.g., conveyors, container sealing robots, loading robots, unloading vehicles, etc.; not shown) configured to place the shipping bins at an outbound location, such as a loading bay.

[0128] In the above explanation of the robot system 100, we have mainly described the state in which the robot system 100 is executing tasks related to the outflow (shipment) of goods. However, when executing tasks related to the inflow or replenishment of goods, basically, this can be handled by creating or obtaining a control sequence for operating the robot system 100 using, for example, machine learning, in the reverse order of the above explanation, and executing the task based on that control sequence.

[0129] [Another application example 2] 10A and 10B are schematic plan views showing another exemplary environment in which a robot system according to an embodiment of the present disclosure can operate, and FIG. 10B is a diagram showing the sorting area 103 in FIG. 10A and its surroundings. A robot system 200 according to this embodiment has a similar configuration to the robot system 100 shown in FIGS. 9A and 9B , except that the robot system 200 stores items directly on the shelves of a rack 15, rather than storing items in a container 16 and then storing the container 16 in a rack. As such, the robot system 200 can be considered an example in which the inspection target in the present disclosure is the item itself, and can be applied to directly storing items in a rack and directly removing items from a rack when the operation target is the item itself in the replenishment process P20 and the retrieval process P30 shown in FIG. 1 .

[0130] That is, the robot system 200 does not need to include the transport robot 12 as the transport unit in the robot system 100, and as a result, the transport robot 11, which accesses the storage area 101 and picks up a rack 15, can move along the first path P1, transport the rack 15 to the sorting area 103, and stop at a stop position SL near the operation robot 13. When the rack 15 arrives in the sorting area 103 in this way and the transport robot 11 stops at the stop position SL as shown in Figures 10A and 10B, each rack 15 functions as a container for dispensing or receiving items.

[0131] The robotic system 200 can then operate the handling robot 13 to, for example, pick up the order items from each rack 15 and transfer them into a shipping container or the like corresponding to the order. Additionally, the robotic system 200 can include one or more units (e.g., conveyors, container sealing robots, loading robots, unloading vehicles, etc.; not shown) configured to place the shipping container or the like at an outgoing location, such as a loading bay.

[0132] According to the robot system 200 configured and controlled as described above, the transport robot 12, the transport area 102, and the second path P2 in the robot system 100 can be omitted, thereby contributing to space saving and improved work efficiency. Note that, also in the above description of the robot system 200, the state in which a task related to the warehousing (shipping) of goods is being executed has been mainly described, but when a task related to the warehousing or replenishment of goods is to be executed, it is possible to deal with this by creating or acquiring, for example by machine learning, a control sequence for operating the robot system 200 in the reverse order of the above description, and executing the task based on that control sequence.

[0133] Although an embodiment of the present disclosure has been described in detail above as an example, the above description merely illustrates an example of the present disclosure in every respect, and is intended to facilitate understanding of the present disclosure and is not intended to limit the present disclosure. Furthermore, it goes without saying that various improvements and modifications can be made without departing from the scope of the present disclosure, and the elements of the embodiments, as well as their arrangement, materials, conditions, shapes, sizes, etc., are not limited to those exemplified or specific ones, and can be changed as appropriate.

[0134] In other words, the embodiments according to the present disclosure do not exclude other configurations, nor do they limit the present disclosure to the above-described embodiments. Various modifications equivalent to the above-described embodiments according to the present disclosure are possible within the scope of the present disclosure. For example, the order of execution of processes, steps, routines, or blocks may be performed in a different order according to alternative implementations within the scope of the present disclosure, and some processes, steps, routines, or blocks may be deleted, moved, added, subdivided, combined, and / or modified within the scope of the present disclosure. Also, each of these processes, steps, routines, or blocks may be performed in a variety of different ways. Furthermore, even if processes, steps, routines, or blocks are performed sequentially in the above-described embodiments, those processes, steps, routines, or blocks may be performed in parallel or non-sequentially at different times. Furthermore, specific numbers set forth herein may have different values ​​or ranges.

[0135] Furthermore, the above-described embodiments are not limited to specific examples of the present disclosure as the "best mode" that can be envisioned, and may be implemented in many alternative ways. Furthermore, the details of the above-described embodiments may vary significantly in their particular implementation and still be encompassed by the technology of the present disclosure. In addition, specific terms used to describe particular features or aspects of the present disclosure are not intended to limit the present disclosure to the specific characteristics, features, or aspects of the present disclosure or to the particular embodiments with which the terms are associated, and therefore the present invention is not limited to any particular meaning except as defined by the claims. Furthermore, while the present invention is defined in any number of claim forms, it goes without saying that various embodiments are contemplated within the scope of the present disclosure.

[0136] This application is based on U.S. Patent Application No. 62 / 792,348, filed on January 14, 2019, the contents of which are incorporated herein by reference. [Explanation of symbols]

[0137] 11, 12...Transport robot, 13, 14...Operation robot, 15...Rack, 16...Operation object, container, 40...Vehicle, 100, 200, 500, 600...Robot system, 101...Storage area, 102...Transport area, 103...Sorting area, 131, 141...End effector, 132, 142...Robot arm, 152...Fall prevention device, 502...Processor, 504...Storage device, 506...Communication device, 508...Input-output device Chair, 510...display, 512...actuating device, 514...transfer motor, 516...sensor, 522...imaging sensor, 524...position sensor, 526...contact sensor, 552...master data, 554...object tracking data, 601...access position, 602...transfer position, 610...control device, 612...data acquisition unit, 614...data storage unit, 616...robot control unit, OL...transfer position, P1...first path, P2...second path, SL...stop position.

Claims

1. A control device for a robot system including an operation robot that operates an operation object and a transport robot that transports the rack in order to store the operation object in a rack and / or take out the operation object from the rack, a data acquisition unit that acquires first data including information about the operation object and information about the rack before the operation object is stored in the rack and / or removed from the rack; a robot control unit that, before storing the operation object in the rack and / or removing the operation object from the rack, selects the rack and transports it to an access position based on the first data, creates or acquires a control sequence for storing the operation object in the rack and / or removing the operation object from the rack, and commands the transport robot to execute a task for transporting the rack to the access position based on the control sequence, and commands the operation robot to execute a task for storing the operation object in the rack and / or removing the operation object from the rack; Equipped with The robot control unit is a control device that commands the operation robot to execute a task of completing the gripping of the operation object and movement of the operation robot to the access position by the operation robot before the rack reaches the access position when storing the operation object in the rack, and / or completing the movement of the operation robot before the rack reaches the access position when removing the operation object from the rack.

2. the data acquisition unit acquires second data including information about the operation target and information about the rack after the operation target is stored in the rack and / or removed from the rack. The control device according to claim 1 .

3. the robot control unit creates or acquires the control sequence based on the second data so that the arrangement of the operation objects on the rack is consolidated. The control device according to claim 2 .

4. further comprising a data storage unit that stores the first data and the second data; The control device according to claim 2 or 3.

5. the data acquisition unit acquires relative position information between the transport robot and the rack; the robot control unit corrects position information of the rack at the access position based on the relative position information, and creates or acquires the control sequence based on the corrected position information.

5. The control device according to claim 1.

6. The rack is configured so that the operation object is stored and / or taken out from the side, and a fall prevention device for the operation object is provided at an end of the rack, the operation robot grasps the operation target from a lateral direction, the robot control unit creates or acquires the control sequence so that the operation target moves beyond the fall prevention device; 6. The control device according to claim 1.

7. the data storage unit stores position information of the plurality of operation objects in the rack for each operation object or each shelf of the rack, as two-dimensional information or three-dimensional information; The control device according to claim 4.

8. the robot system further includes a sensor that captures an image of the rack in which the operation target is stored, the sensor captures an image of an identification code or an identification tag representing identification information of the operation object stored in the rack; 8. A control device according to any one of claims 1 to 7.

9. the robot system further includes a sensor that captures an image of the rack in which the operation target is stored, the sensor captures an image of the rack while the transport robot holding the rack is stationary or while the transport robot holding the rack is moving.

8. A control device according to any one of claims 1 to 7.

10. the robot control unit determines that the putting of the operation object into the rack and / or the taking of the operation object from the rack has been completed based on the operating state of an end effector of the operation robot or tracking of the movement of the operation object before the putting of the operation object into the rack and / or the taking of the operation object from the rack has been completed.

10. A control device according to any one of claims 1 to 9.

11. The data acquisition unit acquires actual measured values ​​or estimated values ​​of information related to the operation target and / or information related to the rack.

11. A control device according to any one of claims 1 to 10.

12. the robot control unit creates or acquires a control sequence that includes specifying a position in the rack to be used for storing the operation object and / or a position of the operation object to be taken out of the rack, based on an image of the operation object and the rack at the access position captured by an image sensor; A control device according to any one of claims 1 to 11.

13. A control device according to any one of claims 1 to 12; a robot system including the operation robot and the transport robot; A logistics system equipped with:

14. A program for causing a computer to function as the control device according to any one of claims 1 to 12.

15. A non-transitory computer-readable recording medium on which the program according to claim 14 is recorded.

16. A control method for a robot system including an operation robot that operates an operation object and a transport robot that transports the rack in order to store the operation object in a rack and / or remove the operation object from the rack, the method being executed using a control device having a data acquisition unit and a robot control unit, the data acquisition unit acquires first data including information about the operation object and information about the rack before storing the operation object in the rack and / or removing the operation object from the rack, the robot control unit, before storing the operation object in the rack and / or removing the operation object from the rack, selects the rack and transports it to an access position based on the first data, creates or acquires a control sequence for storing the operation object in the rack and / or removing the operation object from the rack, and commands the transport robot to execute a task for transporting the rack to the access position based on the control sequence, and commands the operation robot to execute a task for storing the operation object in the rack and / or removing the operation object from the rack, the robot control unit commands the operation robot to execute a task of completing the gripping of the operation object by the operation robot and the movement of the operation object to the access position before the rack reaches the access position when storing the operation object in the rack, and / or completing the movement of the operation robot before the rack reaches the access position when removing the operation object from the rack. Control method.

Images