Picking system operation support device and picking system

Abstract

To provide a picking system operational support apparatus which can plan a flexible picking work according to a picking object or a state of a picking operation part to realize the operation of the picking work in a more efficient picking system.SOLUTION: A classification control device 3 for supporting an operation related to a picking work in a picking system, comprises: a state information acquisition part 4 which acquires state information indicating a state of a workpiece 202 and picking operation parts 2A-2C performing the picking operation of the workpiece 202; a picking success estimation part 5 which estimates success of the picking work based on the state information; and a distribution planning part 6 which generates distribution information 303 indicating an operation plan of the picking work in the picking system based on the estimated success to improve operational efficiency of the picking work in the picking system.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for realizing a picking operation of a picking target (workpiece), and particularly to a technique for supporting more efficient operation of a picking system. [Background technology]

[0002] In recent years, the labor force has been shrinking due to the declining birthrate and aging population, and there is a growing need for automation and autonomy using robots to alleviate labor shortages and improve productivity.In the logistics field in particular, coupled with the recent expansion of the mail order market, there are high expectations for the introduction of autonomous picking robots that use multi-axis robotic arms and other devices to automate the pick-and-place (lifting and transporting) process of workpieces (objects of work) in fixtures, which has traditionally been performed by humans.

[0003] To have a robot autonomously perform picking tasks, including pick-and-place, it is common to recognize environmental information, including the position and orientation of the workpiece, using imaging means installed in the vicinity or on the robot.For example, in the field of factory automation, prior information on the shape and texture of the workpiece is registered in a database, and the position and orientation of the workpiece is estimated by comparing the image acquired by the imaging means with the prior information.

[0004] On the other hand, in the logistics field, represented by e-commerce, a wide variety of work is handled and the work packaging changes with the seasons, so it is desirable to perform picking work without registering information in advance.

[0005] For example, the abstract of Patent Document 1 describes the problem that "it is difficult to properly adhere a suction pad to a package by simply pressing the suction pad against the package," and describes a solution to this problem as including "an input means for inputting an image of the package, and a determination means for determining an area to be adhered to by the suction unit, giving priority to areas of the surface of the packaging material that are unlikely to have wrinkles or holes, based on information relating to the success or failure of suction for each area of ​​the surface of the packaging material identified from the image."

[0006] That is, in Patent Document 1, an area on the surface of a workpiece that is unlikely to have wrinkles or holes is estimated based on an image captured by an imaging device. This allows the position on the image where the workpiece can be picked up to be estimated without having to register any prior information about the workpiece, and in the embodiment, three-dimensional position and orientation information is obtained using a range image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-181687 Summary of the Invention [Problem to be solved by the invention]

[0008] As described above, according to Patent Document 1, it is possible to estimate positions where workpieces can be picked up without registering prior information about the workpieces in advance. However, depending on the layout and cost constraints of the customer's equipment, there are cases where it is not possible to install the desired types of imaging devices and robots in optimal positions on all lines to be introduced, or where it is not possible to prevent the lighting from the customer's equipment from shining on the workpieces.

[0009] When such physical constraints arise, the quality of images (including distance images) captured by the imaging device may deteriorate depending on the line. This can result in, for example, failure to recognize the position and orientation information of the workpiece, or failure to lift the workpiece because the robot is unable to move its hand to the recognized position and orientation, potentially resulting in reduced throughput due to the robot stopping or damage to the workpiece. As described above, Patent Document 1 and other documents have had the problem of difficulty in efficient operation due to physical constraints of the picking system, such as the state of the picking target and the picking device. Therefore, the present invention aims to avoid creating rigid picking work plans and to support efficient operation of the picking system. [Means for solving the problem]

[0010] In order to solve the above problems, the present invention estimates the success or failure of a picking operation based on status information indicating the status of the picking target and the picking unit performing the picking, and adjusts the operation of the picking system including the picking unit, particularly the picking operation, based on the estimated success or failure, thereby improving the operational efficiency of the picking operation of the picking system.

[0011] More specifically, in a picking system operation support device that supports operations related to picking work in a picking system, A plurality of picking operation units included in the picking system, Picking the items to be picked and performing the picking work on the items The plurality of a status information acquisition unit that acquires status information indicating the status of the picking operation unit; and In each of the plurality of picking work units for each of the picking targets a picking success / failure estimation unit that estimates the success or failure of the picking operation; and an allocation planning unit that creates allocation information that indicates an operation plan for the picking operation in the picking system according to the estimated success or failure, the picking success / failure estimation unit creates success / failure estimation information that records the success or failure of each of the plurality of processes of the picking work, and the allocation planning unit removes from the candidates any picking work unit that is indicated as having failed in any of the plurality of processes in the success / failure estimation information, and selects a picking work unit that will perform the picking work from the plurality of picking work units that are not removed from the candidates using the allocation information; The picking system operation support device realizes improvement in operational efficiency regarding picking work in the picking system.

[0012] The present invention also includes a picking system operation support method using the picking system operation support device, and a picking system including the picking system operation support device.The present invention also includes a picking method using a picking system.Furthermore, a program for causing the picking system operation support device to function as a computer and a storage medium for storing the program are also aspects of the present invention. [Effects of the Invention]

[0013] According to the present invention, picking operations can be flexibly planned according to the items to be picked and the state of the picking unit, enabling more efficient operation of picking operations in a picking system. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing a picking area as an example of an application of an embodiment of the present invention. [Figure 2] Functional block diagram of a picking system in Example 1 [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of a control device according to a first embodiment. [Figure 4] Functional block diagram of a picking operation control device in Example 1 [Figure 5] FIG. 10 is a diagram showing details of the processing of the picking success / failure estimation unit in the first embodiment. [Figure 6] Functional block diagram of a workpiece information estimation unit in Example 1 [Figure 7] Functional block diagram of a recognizability estimation unit in the first embodiment [Figure 8] Functional block diagram of an operation ease estimation unit in the first embodiment [Figure 9] FIG. 10 is a diagram showing success / failure estimation information used in the first embodiment. [Figure 10] FIG. 10 is a diagram showing distribution information used in the first embodiment. [Figure 11] FIG. 10 is a diagram for explaining allocation of picking targets in the second embodiment. [Figure 12]System configuration diagram of the picking system in Example 3 [Figure 13] Flowchart showing processing in the third embodiment DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings as appropriate. In this embodiment, it is determined which of a plurality of picking units 2A to 2C in a picking area will perform a picking operation. Here, FIG. 1 is a diagram showing a picking area as an example of an application of the embodiment. A plurality of picking units 2A to 2C are provided in the picking area.

[0016] Furthermore, the fixtures 201 that are placed in the picking area store workpieces 202 that are to be picked. When the fixtures 201 (workpieces 202) are placed in the picking area, they are moved to an appropriate picking work section among the picking work sections 2A to 2C by a sorting device 30 such as a belt conveyor. As a result, the picking work is carried out at the picking work section to which they are moved. Note that the fixtures 201 only need to be able to store the workpieces 202, and their shape and name do not matter. Furthermore, the workpieces 202 are to be picked, and their type does not matter, such as products, merchandise, or parts.

[0017] Furthermore, each line of the picking work units 2A and 2B is provided with hardware for automating the picking work. That is, hardware for performing the picking work, such as line imaging devices 23A and 23B, robot arms 21A and 21B, suction hand 22A, and gripping hand 22B (also simply referred to as hand 22), is provided. A picking work control device 20 that controls this hardware is provided in the picking work units 2A and 2B. Furthermore, in the picking work unit 2C, picking work is performed by workers. Furthermore, a work imaging device 40 is provided at a position where it can capture images of fixtures 201 or works 202 before sorting to each line.

[0018] Here, the line imaging devices 23A, 23B and the workpiece imaging device 40 can be realized by cameras or the like, and capture images of the inserted workpiece 202 and the picking work units 2A, 2B. However, the line imaging devices 23A, 23B and the workpiece imaging device 40 may be installed in positions other than those described above. Furthermore, the line imaging devices 23A, 23B may be provided for each of the multiple picking work units 2A, 2B or each line. Furthermore, although not shown, a line imaging device may be provided in the picking work unit 2C.

[0019] The robot arms 21A and 21B, the suction hand 22A, and the gripping hand 22B are hardware that performs operations for picking work. The robot system including the robot arms 21A and 21B, the suction hand 22A, and the gripping hand 22B includes an autonomous picking robot. The picking work is performed using these pieces of hardware and includes at least one of the processes of recognition, lifting (picking), and transporting (placing). The lifting and transporting operations are also collectively referred to as pick and place, as described below.

[0020] Here, the appropriate picking operation varies depending on the workpiece 202. For example, if a fragile workpiece is gripped too tightly by the suction hand 22A or the gripping hand 22B, it may be damaged. Furthermore, depending on the compatibility between the shape of the workpiece 202 and the shapes of the suction hand 22A or the gripping hand 22B, it may be difficult to lift the workpiece. Furthermore, depending on the size and weight of the workpiece 202, it may be difficult to move the workpiece along the line, or it may take a long time for the robot arms 21A and 21B to move the workpiece. In other words, depending on the condition of the workpiece and the picking operation unit, the throughput of the picking operation may decrease. Furthermore, even in the picking operation unit 2C where picking is performed manually, the efficiency of the picking operation may decrease if there are few workers due to breaks or other reasons.

[0021] As described above, the efficiency of the picking operation decreases depending on the state of the workpiece 202 to be picked and the picking units 2A to 2C. Therefore, in this embodiment, the picking system 100 can more efficiently operate the picking operation by flexibly responding to the state of the workpiece 202 and the picking units 2A to 2C. The state of the workpiece 202 includes the characteristics (hardness, size, shape, weight) and quantity of the workpiece 202. The state of the picking units 2A to 2C includes the configuration, work capacity, characteristics, robot specifications, and quantity of the picking units. The robot specifications include the operation program, hand type, etc.

[0022] Therefore, in this embodiment, a sorting control device 3 that supports operations related to picking work in a picking system 100 includes a status information acquisition unit 4 that acquires status information indicating the status of a work 202 and the picking work units 2A to 2C that perform picking work on the work 202, a picking success / failure estimation unit 5 that uses the status information to estimate the success or failure of the picking work, and an allocation planning unit 6 that creates allocation information 303 that indicates an operation plan for the picking work in the picking system according to the estimated success or failure, thereby improving the operational efficiency of the picking work in the picking system 100. Below, examples showing specific contents of this embodiment will be described. [Example]

[0023] (Overall composition) 2 is a functional block diagram of a picking system 100 in the first embodiment. The picking system 100 in the present embodiment includes a control device 1, a picking operation unit group 2, and a sorting control device 3, which are connected to each other via a communication path such as a network. First, the control device 1 outputs a control command to control the operation of a picking operation unit that performs a corresponding picking operation among the picking operation units (2A to 2C) included in the picking operation unit group 2, and controls them.

[0024] The picking operation unit group 2 is made up of picking operation units (2A to 2C) that perform picking operations. The configuration of each of the three picking operation units 2A, 2B, and 2C in this embodiment will be described later. The sorting control device 3 creates sorting information that indicates an operation plan for the picking operation according to the status of the work and the picking operation units 2A to 2C. To this end, the sorting control device 3 has a status information acquisition unit 4, a picking success / failure estimation unit 5, an allocation plan unit 6, a work instruction generation unit 7, and a memory unit that stores the following information. The sorting control device 3 is an example of a picking system operation support device.

[0025] Here, each piece of information stored in the storage unit will be explained. First, picking work history information 300 is information showing the history (achievement) of past picking work by each picking work unit 2A to 2C. Furthermore, work unit data 301 is data showing the configuration and parameters of each picking work unit 2A to 2C. Furthermore, success / failure estimation information 302 is information showing the success / failure of picking work in each picking work unit 2A to 2C, and includes, for example, the success / failure of realizing each work process and the picking work time. Note that the success / failure estimation information 302 may be information for each fixture that stores workpieces as the picking work units 2A to 2C. Furthermore, allocation information 303 is information showing an operation plan for picking work in the picking system 100. The allocation information 303 indicates which picking work unit 2A to 2C should perform the picking work for the workpiece. In other words, it is used to sort the destinations of the workpieces and fixtures. More specifically, the allocation information 303 includes the type, number, and scheduled time of workpieces to be picked by each of the picking units 2A to 2C. It is desirable that the allocation information 303 also includes the actual time of the picking work performed by the picking units 2A to 2C. This concludes the overview of each piece of information, but the specific contents of the success / failure estimation information 302 and the allocation information 303 will be explained later.

[0026] Below, we will explain each functional block of the sorting control device 3. First, the status information acquisition unit 4 acquires the status of the picking target and the picking work unit where the picking work is performed for that target. Here, the picking target includes at least the work 202 to be input into the picking system 100, and may also include the fixtures 201 that store the work. Furthermore, the picking work unit includes picking work units 2A to 2C.

[0027] The status information acquisition unit 4 then acquires the status of the workpiece 202 and the fixtures 201 as follows: First, the status information acquisition unit 4 acquires imaging data of the workpiece 202 and the fixtures 201 from a workpiece imaging device 40 such as a camera. Here, the workpiece imaging device 40 is installed at the front stage of each line. As a result, the workpiece imaging device 40 can capture images of at least a portion of the workpiece 202 and the fixtures 201 before sorting. For this reason, the workpiece imaging device 40 can be realized by various cameras such as a stereo camera, a structured light camera, a time-of-flight camera, or a monocular camera, but the type is not limited. Then, the status information acquisition unit 4 performs image processing on the imaging data acquired from the workpiece imaging device 40 to create work status information indicating the status of the workpiece 202.

[0028] Furthermore, the status information acquisition unit 4 creates work status information including the specifications of each picking operation unit 2A to 2C from the operation unit data 301. Here, the specifications of the picking operation units 2A to 2C indicate performance in picking operations, and may include the arrangement of the robots 21A and 21B, the range of motion, the type of suction hand 22A and gripping hand 22B, the operating speed, and the workpieces that can be handled (size, shape), etc.

[0029] The status information acquisition unit 4 can then create status information including work status information and working unit status information. This status information can take any form as long as it can be used to estimate the success or failure of the picking operation, which will be described later.

[0030] Furthermore, the status information may be created within the sorting control device 3, or may be created in cooperation between the sorting control device 3 and the picking operation control device 20. In this case, the sorting control device 3 creates work status information for the work 202, and the picking operation control device 20 creates work unit status information for the picking operation units 2A to 2C, and this can be achieved by merging these.

[0031] Furthermore, the picking success / failure estimation unit 5 uses the status information to estimate the success or failure of the picking operation at at least one of the picking units 2A to 2C. That is, the picking success / failure estimation unit 5 uses the status information to create success / failure estimation information 302. At this time, it is desirable that the picking success / failure estimation unit 5 also use at least one of the picking operation history information 300 and the operation unit data 301. Here, by using the picking operation history information 300, the success / failure estimation information 302 that more accurately reflects the actual situation based on the actual results is created. Furthermore, by using the operation unit data 301, the success / failure estimation information 302 that reflects the actual situation based on the specifications of each of the picking units 2A to 2C is created.

[0032] Furthermore, the allocation planning unit 6 creates allocation information 303 depending on the success or failure of the picking work. Here, creating the allocation information 303 includes correcting and changing the allocation information 303 that was created in the past.

[0033] Furthermore, the work instruction generation unit 7 generates a work instruction for controlling the picking work for the workpiece to be performed by one of the picking work units 2A to 2C based on the allocation information 303, and outputs the work instruction to the control device 1.

[0034] In the following, the control device 1, the picking operation group 2, the picking operation history information 300, the operation unit data 301, the picking success / failure estimation unit 5, the allocation planning unit 6, and the picking execution unit 10 will be described in detail.

[0035] (Control device 1) The control device 1 receives work instructions created by the work instruction generation unit 7 and outputs control commands corresponding to the work instructions to the sorting device 30 and the picking units 2A to 2C (particularly the picking units 2A and 2B). As a result, picking work according to the work instructions can be performed using the sorting device 30 and the picking units 2A to 2C. The control device 1 will be described in detail below. FIG. 3 is a diagram showing the hardware configuration of the control device 1 in this embodiment. As shown in FIG. 3, the control device 1 has a CPU (Central Processing Unit) 11, a bus 12, a ROM 13 (Read Only Memory), a RAM 14 (Random Access Memory), a storage device 15, a network I / F 16, an imaging I / F 17, a screen display I / F 18, and an input I / F 19, which are connected via the bus 12.

[0036] First, the CPU 11 controls the overall operation of the control device 1. Furthermore, the bus 12 transmits commands from the CPU 11 to other parts. The imaging I / F 17 connects to the work imaging device 40 and the line imaging devices 23A and 23B. The screen display I / F 18 has a function for performing screen output and is connected to the screen. Note that the screen display I / F 18 can be omitted. Furthermore, the input I / F 19 accepts external input from an external device. In this way, the control device 1 can be configured with a general computer. Furthermore, the control device 1 can also be configured with a PLC (Programmable Logic Controller).

[0037] The storage device 15 also stores a program 15A for executing each function, an OS 15B, a three-dimensional model 15C, parameters 15D such as a database, and the like.

[0038] Also connected to the network I / F 16 is a robot control device 16A for controlling and operating the picking work units 2A, 2B, particularly the robots. Furthermore, connected to the network I / F 16 is a sorting device control device 16B for controlling and operating the sorting device 30. The robot control device 16A and the sorting device control device 16B output control commands to their controlled objects. These control commands are generated by the CPU 11 in accordance with the program 15A based on the work instructions. The control device 1 may be implemented as a single device together with the sorting control device 3, or may be provided for each picking work unit. This concludes the explanation of the control device 1.

[0039] (Configuration of the picking operation control device 20, workpiece imaging device 40, and sorting device 30) Next, the picking operation control device 20, workpiece imaging device 40, and sorting device 30 will be described. These are installed as shown in FIG. 3A. Then, the workpiece imaging device 40 images the workpiece 202 and the fixture 201 storing it to obtain imaging data. Then, image processing is performed on the imaging data to create workpiece status information indicating the status of the workpiece 202. Then, based on this workpiece status information, the sorting device 30 is controlled to move the fixture 201 to a picking operation unit suitable for the workpiece 202. In other words, the workpiece 202 is sorted to a picking operation unit according to its own status. Then, at the picking operation unit to which the workpiece 202 has been moved, a picking operation is performed on the workpiece 202.

[0040] The picking unit 2A has a robot arm 21A (hereinafter also referred to as "robot"), and a suction hand 22A and a line imaging device 23A (hereinafter also referred to as "hand-eye camera") attached to the hand of the robot 21A. The picking unit 2B has a robot 21B, and a gripping hand 22B attached to the hand of the robot 21B. It also has a line imaging device 23B (hereinafter also referred to as "fixed camera") installed independently of the robot 21B. Furthermore, in the picking unit 2C, a worker performs picking work. As described above, the configurations of the picking units 2A to 2C are different, and the efficiency and success of the picking work, such as whether it is possible to perform the work, change depending on the state of the workpiece 202.

[0041] Next, the picking operation control device 20 will be described. This is a device that controls hardware including robots 21A and 21B. FIG. 4 is a functional block diagram of the picking operation control device 20 in this embodiment. In FIG. 4, the picking operation control device 20 has an item recognition unit 25, a trajectory planning unit 26, a robot control unit 27, a hand control unit 28, and a database that stores a recognition algorithm 304.

[0042] First, the recognition algorithm 304 estimates, as an item recognition result, one or more of the size, position and orientation of the workpiece 202 and fixture 201, and the positions on the surface of the workpiece 202 to be picked up by the suction hand 22A and the gripping hand 22B. For this purpose, one or more of two-dimensional images of the workpiece 202 and the fixture 201 storing it and three-dimensional point clouds (including parallax images and distance images) are used.

[0043] The item recognition unit 25 also acquires image data consisting of one or more of a two-dimensional image and a three-dimensional point cloud of the workpiece 202 using the line imaging devices 23A and 23B. The item recognition unit 25 then estimates the item recognition result using a recognition algorithm 304. That is, it creates item recognition information. In response to the control command described above, the trajectory planning unit 26 creates trajectory information indicating the trajectory of the transport operation of the picking operation using the item recognition information and the hardware specifications of the picking operation units 2A and 2B indicated by the operation unit data 301 described later. Examples of the hardware specifications of the picking operation units 2A and 2B include the type, arrangement, range of motion, speed, and acceleration parameters. The trajectory information also indicates a series of trajectories from lifting the workpiece 202 to transporting it. That is, it indicates the trajectory from the lifting operation to the transport operation.

[0044] Furthermore, the robot control unit 27 actually controls the robots 21A and 21B based on the trajectory information. That is, the robot control unit 27 generates control signals to control the robots 21A and 21B. Furthermore, the hand control unit 28 uses the item recognition information in response to the control command described above to control the suction hand 22A and the gripping hand 22B when the robots 21A and 21B move to positions where they can grip the workpiece 202. As a result, the suction hand 22A and the gripping hand 22B can pick up or grip the workpiece 202. To achieve this, the hand control unit 28 generates control signals to control the suction hand 22A and the gripping hand 22B. As described above, the picking operation control device 20 performs a picking operation on the workpiece 202 in response to the control command. Here, the control command is based on the received allocation information 303. Therefore, according to this embodiment, a picking operation in accordance with the allocation information 303 can be realized.

[0045] Here, the line imaging device 23B of the picking unit 2B employs a fixed camera system. While the robots 21A and 21B are transporting the workpiece 202, the next workpiece 202 can be imaged and recognized. Therefore, the picking unit 2B is characterized by its excellent takt time. In other words, it can complete picking operations in a short time. However, because the viewpoint is fixed, if there is ambient lighting 25B in the picking area of ​​the picking unit 2B, as shown in FIG. 1, the quality of the image data may be reduced due to effects such as overexposure of the workpiece in the image caused by the lighting. Furthermore, depending on the installation location of the line imaging device 23B, it may be far from the workpiece 202. Furthermore, if a stereo camera or structured light camera, which reduces the accuracy of 3D measurement as the imaging distance increases, is used as the line imaging device 23B, the measurement performance required for recognizing small workpieces, which require high-precision imaging, may be insufficient.

[0046] On the other hand, the line imaging device 23A of the picking operation unit 2A uses a hand-eye camera system. In this case, the robot 21A must move once to a shooting position to photograph the workpiece 202. This has the advantage of reducing takt time compared to the fixed camera system. However, the viewpoint during photography can be flexibly changed. Therefore, even if the workpiece in the image is blown out due to the customer's ambient lighting 25A, the line imaging device 23A can perform imaging with reduced effects of blown out highlights by changing the viewpoint. Furthermore, since the line imaging device 23A can photograph the workpiece 202 from a close distance, it is easy to achieve high-precision 3D measurement.

[0047] In this embodiment, a fixed camera type picking unit 2B that places importance on takt time is used in combination with a hand-eye camera type picking unit 2A that has a slower takt time. This is because the picking area near the picking unit 2A is difficult to install a fixed camera due to the presence of peripheral installations 24A (installations in the customer's environment).

[0048] Furthermore, the picking work units 2A and 2B use different suction and gripping methods for the suction hand 22A and gripping hand 22B. In this embodiment, the picking work unit 2B uses a gripping hand that can stably grip the workpiece 202 even when it is moved at high speed. On the other hand, the picking work unit 2A has a hand-eye camera installed on the robot, which reduces the payload capacity of the robot 21A. For this reason, the picking work unit 2A uses a lighter suction hand to ensure the payload capacity of the workpiece.

[0049] As described above, the capabilities of each picking unit may differ depending on the environment of the picking area and the requirements for the picking work. As a result, the types of workpieces 202 that each picking unit 2A, 2B can handle or is good at handling vary. Therefore, it is desirable to select a picking unit (line) that will handle the workpieces 202. Therefore, in this embodiment, a picking unit that will perform the picking work is selected depending on the status of the workpieces 202 and each picking unit 2A, 2B. This will be explained in more detail later.

[0050] The number of picking units and components (hardware) in the above-described embodiment are merely examples. Therefore, the line imaging devices 23A and 23B may use a hand-eye camera and a fixed camera in combination in one picking unit, and the hand may be capable of both suction and gripping, and are not limited to the above example.

[0051] The picking operation control device 20 can be configured using a general computer or the above-mentioned PLC. Furthermore, the picking operation control device 20 can be configured integrally with the control device 1 or the sorting control device 3. In other words, the picking operation control device 20 can be configured integrally with the control device 1, the sorting control device 3, or the control device 1 and the sorting control device 3.

[0052] (Picking work history information 300) The picking operation history information 300 is information showing the history of past picking operations performed by each of the picking operation units 2A to 2C. It is desirable that the picking operation history information 300 include one or more of the following: Image data used in the item recognition unit 25. The algorithm and parameters used among the recognition algorithms 304. The item recognition result is one or more of the size, position and orientation of the workpiece 202 and the fixture 201, and the position on the workpiece surface where the hand picks up the workpiece. The time required for recognition, the robot trajectory generated by the trajectory planning unit 26, the time required for trajectory generation, and the time required for movement. Control parameters used in the hand control unit 28.

[0053] (Working Unit Data 301) The operation unit data 301 is data indicating the configuration and parameters of each of the picking operation units 2A to 2C. The operation unit data 301 indicates the status (characteristics, specifications, etc.) of each picking operation unit. For this reason, it is desirable that the operation unit data 301 include, for example, one or more of the following: Number of picking units. The configuration of each picking work unit includes specifications such as the type, size, range of motion, speed, and acceleration of the robot 21, and its installation location. Specifications such as the type, size, range of motion, payload capacity, payload size, and payload material of the hand 22, and the attachment position relative to the robot. Specifications such as the type of line imaging device 23, imaging method, resolution, angle of view, focal length, blur, measurement distance, 3D measurement accuracy, measurable materials, and, in the case of an active measurement camera, light projection intensity. Mounting position relative to the robot. Shape and location of surrounding installation 24A. Type, position, light intensity and half-value angle of peripheral lighting 25A and 25B.

[0054] Here, design values ​​can be used for the attachment positions of the hand 22 and the line imaging devices 23A and 23B relative to the robot 21. Alternatively, values ​​estimated by a method for estimating the hand shape called tool calibration or a method for estimating the imaging position called hand-eye calibration may be used. Furthermore, the portable materials of the hand 22 and the 3D measurement accuracy and measurable materials of the line imaging device 23 may be evaluated in advance and registered in the working unit data 301.

[0055] (Picking success / failure estimation unit 5) The picking success / failure estimation unit 5 estimates the success or failure of the picking operation of the work 202 in the fixtures 201, which are the objects to be picked in each picking operation unit. More specifically, the picking success / failure estimation unit 5 creates success / failure estimation information 302 indicating the success or failure of the picking operation. This success / failure estimation information 302 preferably includes feasibility information and operation time information for each operation process in the picking operation. The processing of the picking success / failure estimation unit 5 will be described in detail below.

[0056] FIG. 5 is a diagram showing details of the processing of the picking success / failure estimation unit 5 in this embodiment. In FIG. 5, the picking success / failure estimation unit 5 has a work information estimation unit 51, a recognition ease estimation unit 52, and an operation ease estimation unit 53. First, the work information estimation unit 51 creates work information 3021 indicating the state of the work 202 to be picked, using the state information from the state information acquisition unit 4, the picking work history information 300, and the work unit data 301. The work information 3021 may also include the state of the fixture 201. It is also desirable to use work state information based on imaging data captured by the work imaging device 40 as the state information.

[0057] Furthermore, the recognizability estimation unit 52 uses the state information, picking work history information 300, and work unit data 301 to create recognizability information 3022 indicating the ease of recognition of the workpiece 202 by the line imaging devices 23A, 23B and the recognition algorithm 304. Here, it is desirable that the recognizability information 3022 is information indicating whether recognition with the accuracy required for picking work is possible in each of the picking work units 2A, 2B. It is also desirable that the recognizability estimation unit 52 use workpiece state information as the state information.

[0058] Furthermore, the operation ease estimation unit 53 generates operation ease information 3023 indicating whether the workpiece 202 can be picked and placed while avoiding the fixture 201 and peripheral installation object 24A using a combination of the state information, the robots 21A and 21B, the suction hand 22A, and the gripping hand 22B. For this purpose, the operation ease estimation unit 53 uses the picking work history information 300 and the working unit data 301. Here, pick and place refers to a lifting operation (e.g., grasping / sucking) and a transporting operation / process in a picking operation. In addition, it is desirable that the operation easiness estimation unit 53 uses work state information as the state information.

[0059] The following describes in detail the work information estimation unit 51 and the work information 3021, the recognizability estimation unit 52 and the recognizability information 3022, and the operation easiness estimation unit 53 and the operation easiness information 3023. Note that the operation easiness information 3023 is information indicating the easiness of the lifting operation and the transporting operation, but lifting operation easiness information and transporting operation easiness information for the lifting operation and the transporting operation, respectively, may also be used.

[0060] (Work information estimation unit 51 and work information 3021) First, the workpiece information estimation unit 51 creates workpiece information 3021 that indicates the state of the workpiece 202. Here, Fig. 6 is a functional block configuration diagram of the workpiece information estimation unit 51 in this embodiment. In Fig. 6, the workpiece information estimation unit 51 has an imaging data acquisition unit 510, a workpiece pattern estimation unit 511, a workpiece shape estimation unit 512, a workpiece material estimation unit 513, a workpiece reflectance estimation unit 514, a workpiece stacking method estimation unit 515, a workpiece weight estimation unit 516, a workpiece number estimation unit 517, a fixture shape estimation unit 518, and a past handling presence / absence estimation unit 519.

[0061] First, the imaging data acquisition unit 510 acquires imaging data from the work imaging device 40 and creates work status information for the work 202 and the fixtures 201. This is similar to (part of) the function of the status information acquisition unit 4. Therefore, the imaging data acquisition unit 510 may be omitted, or the corresponding function of the status information acquisition unit 4 may be omitted. Note that each component of the work information estimation unit 51 described below executes processing using such status information.

[0062] Furthermore, the workpiece pattern estimation unit 511 estimates the presence or absence of a characteristic pattern that can be used for item recognition on the surface of the workpiece 202. Furthermore, the workpiece shape estimation unit 512 estimates the shape and size of the workpiece 202. Furthermore, the workpiece material estimation unit 513 estimates the material of the workpiece 202. Furthermore, the workpiece reflectance estimation unit 514 estimates the reflectance of the workpiece 202. Furthermore, the workpiece stacking manner estimation unit 515 estimates the manner in which the workpieces 202 are stacked within the fixtures 201.

[0063] Furthermore, the workpiece weight estimation unit 516 estimates the weight of the workpieces 202. Here, the weight may be the weight of each individual workpiece 202 and / or the total weight of the workpieces 202 in the fixture 201. Furthermore, the workpiece number estimation unit 517 estimates the number of workpieces 202 in the fixture 201. Furthermore, the fixture shape estimation unit 518 estimates the shape of the fixture 201. Furthermore, the past handling presence / absence estimation unit 519 refers to the picking work history information 300 and determines whether the workpiece 202 to be picked has been handled in the past.

[0064] Hereinafter, details of the workpiece pattern estimation unit 511 to the past handling presence / absence estimation unit 519 will be described. First, the workpiece pattern estimation unit 511 extracts edge information of the brightness gradient for each pixel region from the image data or workpiece state information of the workpiece 202, for example, a two-dimensional image. Then, the workpiece pattern estimation unit 511 estimates the presence or absence of a characteristic pattern that can be used for item recognition on the workpiece surface based on whether or not the density of the edge information exceeds a threshold value.

[0065] The workpiece shape estimation unit 512 estimates the shape and size of the workpiece 202 by combining, for example, edge information with gradient information of the point cloud, called the curvature of the three-dimensional point cloud. Alternatively, the recognition algorithm 304 of the picking operation control device 20 or a similar algorithm may be applied to the workpiece state information or the image data.

[0066] Furthermore, the workpiece material estimation unit 513 can estimate the material by, for example, comparing the pattern for each region on the two-dimensional image of the workpiece state information or the imaging data with a template image for each material that has been prepared in advance. Furthermore, when the curvature of the three-dimensional point cloud matches a predetermined pattern, the workpiece material estimation unit 513 may estimate the material at a granularity level, such as that it is a soft, amorphous material (such as vinyl).

[0067] Furthermore, the workpiece reflectance estimation unit 514 can estimate the reflectance from the change in brightness on the image corresponding to the degree of dimming by installing, for example, a dimmable light in the workpiece imaging device 40. The workpiece reflectance estimation unit 514 may also estimate the reflectance for each region from the change in brightness for each region of the image when multiple lights are projected from different directions. Note that the lighting described here may be the above-mentioned ambient lighting 25A and 25B.

[0068] Furthermore, the workpiece packing manner estimation unit 515 and the workpiece number estimation unit 517 apply, for example, the recognition algorithm 304 or a similar algorithm to the actual image data captured by the workpiece imaging device 40 or the workpiece state information based thereon. This makes it possible to estimate the stacking manner and number of the workpieces 202.

[0069] Furthermore, the workpiece weight estimation unit 516 can estimate the weight based on, for example, the weight measured by a weighing scale placed at a position where the fixture 201 is photographed by the workpiece imaging device 40. In other words, the weight of each workpiece 202 can be estimated from the measured weight and information on the volume and number of the workpieces 202 within the fixture 201 from the imaging data by the workpiece imaging device 40 or workpiece status information based thereon.

[0070] In addition, the fixture shape estimation unit 518 can recognize the edges of the fixture 201 from a two-dimensional image, for example, when the fixture 201 is photographed by the work imaging device 40, and estimate the three sides of the fixture by calculating the coordinates of the three-dimensional point cloud at the edges.

[0071] Furthermore, the past handling presence / absence estimation unit 519 can confirm whether or not the workpiece has been handled by, for example, template matching to determine whether the pattern estimated by the workpiece pattern estimation unit 511 appears in the image of the history of the picking work history information 300. Furthermore, the past handling presence / absence estimation unit 519 may extract and compare image feature amounts of the pattern portion (template matching).

[0072] Although an example of creating the work information 3021 has been shown above, the creation is not limited to this example. For example, the work shape, material, softness, reflectance, stacking method, weight, number, and fixture shape may be estimated using a neural network that inputs image data captured by the work imaging device 40 and work state information based on the image data. For this purpose, the neural network may be trained in advance.

[0073] In the configuration of this embodiment, for example, if a structured light camera using triangulation is used as the workpiece imaging device 40, installing it near the fixture 201 makes it possible to measure a highly accurate 3D point cloud. Therefore, the shapes of the workpiece 202 and the fixture 201 can also be measured with high accuracy. Furthermore, by taking the time to create workpiece information using multiple recognition algorithms in the previous stage, more accurate estimation becomes possible. As described above, it is possible to create workpiece information 3021, which is information indicating the state of the workpiece 202 and / or the fixture 201 that stores it.

[0074] The work information estimation unit 51 may omit creating the work information 3021. In this case, the work information 3021 may be manually input, or may be created in advance and stored in the storage unit. This includes using the work information 3021 included in the picking work schedule.

[0075] (Recognizability estimation unit 52) Next, the recognizability estimation unit 52 creates recognizability information 3022 using the image data captured by the work imaging device 40 or work state information based thereon, the picking work history information 300, and the working unit data 301. Here, the recognizability information 3022 is information indicating whether the work 202 can be recognized by the line imaging devices 23A, 23B and the recognition algorithm 304 to the extent that picking work can be performed.

[0076] 7 is a functional block diagram of the recognizability estimation unit 52 of this embodiment. In FIG. 7, the recognizability estimation unit 52 includes an imaging data acquisition unit 520, a work information acquisition unit 521, a working unit information acquisition unit 522, a past state information acquisition unit 523, a two-dimensional measurement performance estimation unit 524, a three-dimensional measurement performance estimation unit 525, an article recognition information estimation unit 526, and a recognition time estimation unit 527.

[0077] First, the imaging data acquisition unit 520 acquires imaging data from the work imaging device 40 and creates work status information for the work 202 and the fixtures 201. This is similar to (part of) the function of the status information acquisition unit 4. Therefore, the imaging data acquisition unit 520 may be omitted, or the corresponding function of the status information acquisition unit 4 may be omitted. In this way, since the imaging data acquisition unit 520 has the same function as the imaging data acquisition unit 510, either one may be omitted.

[0078] Furthermore, the workpiece information acquisition unit 521 acquires workpiece information 3021. Furthermore, the working unit information acquisition unit 522 acquires the status of the line imaging devices 23A, 23B from the working unit data 301. This status includes at least one of the type, imaging method, resolution, angle of view, focal length, blur, imaging distance, 3D measurement accuracy, and measurable materials of the line imaging devices 23A, 23B. Furthermore, if the line imaging devices 23A, 23B are active measurement type cameras, the status also includes specifications such as the light projection intensity, the mounting positions relative to the robots 21A, 21B, and at least one of the type, position, light intensity, and half-value angle of the peripheral lighting 25A, 25B.

[0079] Furthermore, the past state information acquisition unit 523 acquires past item recognition information by the picking work units 2A and 2B of the workpieces 202 that have been handled (picked) in the past from the workpiece information 3021. Furthermore, the two-dimensional measurement performance estimation unit 524 estimates the quality of the two-dimensional image obtained when the workpiece 202 is imaged by the line imaging devices 23A and 23B in each of the picking work units 2A and 2B. Furthermore, the three-dimensional measurement performance estimation unit 525 estimates the quality of the three-dimensional point cloud (or "distance image," "parallax image," "three-dimensional mesh," etc.) obtained when the workpiece 202 is imaged by the line imaging devices 23A and 23B in each of the picking work units 2A and 2B.

[0080] The item recognition information estimation unit 526 estimates one or more of the detection rate, size estimation error, normal estimation error, and gripping point estimation error in item recognition based on the quality of the estimated 2D image and 3D point cloud. The recognition time estimation unit 527 estimates the item recognition time in each of the picking work units 2A and 2B.

[0081] Hereinafter, the two-dimensional measurement performance estimation unit 524 and the three-dimensional measurement performance estimation unit 525 will be described in detail.

[0082] First, the 2D measurement performance estimation unit 524 estimates the quality of the 2D image, for example, from the specifications and placement information of the line imaging devices 23A, 23B and ambient lighting 25A, 25B in each picking operation unit 2A, 2B, and the work information 3021. Here, this quality includes image blur information based on blur at the imaging distance, and the degree of overexposure based on the intensity of the lighting and the color and reflectance of the work. Furthermore, the 2D measurement performance estimation unit 524 uses a simulator to generate images that would be virtually captured by the line imaging devices 23A, 23B by combining the work information 3021 and operation unit data 301. Then, the item recognition information estimation unit 526 and the recognition time estimation unit 527 can generate information on whether an item can be recognized and information on the recognition time using a recognition algorithm such as template matching using a pattern in the recognition algorithm 304.

[0083] Similarly, the 3D measurement performance estimation unit 525 estimates the quality of the 3D image using, for example, the specifications and placement information of the line imaging devices 23A and 23B and the ambient lighting 25A and 25B in each picking operation unit 2A and 2B, and work information 3021. This quality includes at least one of the following: the distance error of the 3D point cloud on the work surface at the shooting distance, the standard deviation of the 3D point cloud, the missing ratio, and the expansion rate. Furthermore, the item recognition information estimation unit 526 and the recognition time estimation unit 527 can use a simulator to generate a 3D point cloud that would be captured virtually by the line imaging devices 23A and 23B by combining the work information 3021 and information from the operation unit database. Then, information such as the detection rate, size estimation error, normal estimation error, gripping point estimation error, and recognition time when an item is recognized by the recognition algorithm of the recognition algorithm 304 is estimated.

[0084] Although the above describes an example of a method for creating the recognizability information 3022, the method is not limited to this example. For example, the recognizability information 3022 may be created using a neural network that receives image data captured by the workpiece imaging device 40 as input. For this purpose, the neural network may be trained to classify the workpieces 202 as recognizable or unrecognizable at each picking unit 2A, 2B, and then used for inference during actual operation. The neural network may also be trained to additionally learn the picking operation history information 300 to change the inference criteria. Alternatively, for a workpiece 202 determined by the past handling presence / absence estimation unit 519 to have been handled in the past, part of the result may be estimated as the recognizability information. In addition, if the line imaging devices 23A, 23B are hand-eye cameras, the recognizability information may be estimated after changing the imaging position multiple times on a simulator.

[0085] As described above, before picking work is performed, it is possible to estimate in advance whether or not the work 202 and the fixture 201 storing it can be recognized by each of the picking work sections 2A and 2B.

[0086] (Easy-to-operate estimation unit 53) The operation ease estimation unit 53 creates operation ease information 3023 indicating whether the combination of the robots 21A, 21B, the suction hand 22A, and the gripping hand 22B can pick and place the workpiece 202 while avoiding the fixture 201 and the peripheral installation object 24A. For this purpose, the operation ease estimation unit 53 uses the image data captured by the workpiece imaging device 40 or workpiece state information based thereon, the picking work history information 300, and the working unit data 301.

[0087] 8 is a block diagram of the operation ease estimation unit 53 in this embodiment. In Fig. 8, the operation ease estimation unit 53 has an imaging data acquisition unit 530, a work information acquisition unit 531, a working unit information acquisition unit 532, a past operation information acquisition unit 533, a pick trajectory feasibility estimation unit 534, a grip easiness estimation unit 535, a pickup easiness estimation unit 536, a place trajectory feasibility estimation unit 537, and an operation time estimation unit 538.

[0088] First, the imaging data acquisition unit 530 has the same functions as the imaging data acquisition units 510 and 520. Therefore, at least one of these can be omitted. Furthermore, the work information acquisition unit 531 has the same functions as the work information acquisition unit 521. Therefore, it is also possible to omit one of these. Furthermore, the operation unit information acquisition unit 532 acquires the configuration of each picking operation unit 2A, 2B from the operation unit data 301. Here, the operation unit information acquisition unit 532 may create the configuration of the picking operation units 2A, 2B based on the imaging data from the line imaging devices 23A, 23B and work status information based thereon.

[0089] Furthermore, the past operation information acquisition unit 533 acquires past operation information of the picking units 2A and 2B for the workpieces 202 that were previously handled (picked) from the workpiece information 3021. This operation information indicates the operation of the pick-and-place process.

[0090] Furthermore, the pick trajectory feasibility estimation unit 534 determines whether it is possible to generate a robot trajectory that allows the robots 21A, 21B, suction hands 22A, and gripping hands 22B to avoid physical interference with the fixture 201 and peripherally installed object 24A and to pick up or grip the workpiece 202. Furthermore, the grip easiness estimation unit 535 and the suction easiness estimation unit 536 estimate whether the workpiece 202 on the fixture 201 can be gripped / sucked by the suction hands 22A and gripping hands 22B. "Can be gripped / sucked" is merely an example, and it is sufficient to estimate whether the lifting operation is possible.

[0091] Furthermore, the place trajectory feasibility estimation unit 537 determines whether it is possible to generate a robot trajectory that enables the robots 21A, 21B, the suction hand 22A, and the gripping hand 22B to avoid physical interference with the fixture 201 and the peripheral installation 24A and to carry out the transport operation for the workpiece 202. Furthermore, the operation time estimation unit 538 estimates the series of operations described above, that is, the time required for pick and place.

[0092] These units, pick trajectory success / failure estimation unit 534 to operation time estimation unit 538, estimate the success or failure of picking operations such as lifting operations and moving operations based on operation unit status information indicating the status of picking operation units 2A and 2B. Details of this will be explained below.

[0093] First, the pick trajectory feasibility estimation unit 534 and the place trajectory feasibility estimation unit 537 reproduce the workpiece 202 and its fixture 201 at the picking positions of the picking units 2A and 2B on a 3D simulator. The workpiece information 3021 is used for this purpose. For this purpose, the pick trajectory feasibility estimation unit 534 and the place trajectory feasibility estimation unit 537 use related information, including the positions of the robots 21A and 21B, the suction hand 22A, the gripping hand 22B, the line imaging devices 23A and 23B, and the peripheral installation object 24A. The size of the fixture 201, the weight, position, and orientation of the workpiece 202, etc., may also be used. This allows the trajectory planning unit 26 to determine whether the trajectory of the lifting operation and the trajectory of the transport configuration for the picking and placing in the picking operation can be generated. As a result, the operation time estimation unit 538 can estimate the execution time of the corresponding pick and place.

[0094] The above-described pick trajectory feasibility estimation unit 534 and place trajectory feasibility estimation unit 537 generate success / failure estimation information 302 indicating whether the pick trajectory and place trajectory of each of the picking operation units 2A and 2B can be generated and the success / failure of their execution times. At this time, the work unit status information of the work information 3021 is used.

[0095] Furthermore, each of the gripping ease estimation unit 535 and the suction ease estimation unit 536 compares the workpiece information 3021 and the working unit data 301 to estimate whether the workpiece 202 can be gripped or picked up. For this purpose, the workpiece shape, material, and position and orientation in the workpiece information 3021 are used. Furthermore, workpieces that can be handled by the suction hand 22A and the gripping hand 22B are used as the working unit data 301. Note that the gripping ease estimation unit 535 and the suction ease estimation unit 536 may be realized in a single configuration as a picking ease estimation unit. These units only need to be able to estimate whether the workpiece can be gripped or picked up as long as they can estimate whether a lifting operation is possible.

[0096] The estimation of the operation ease information 3023 is not limited to the above-mentioned method. For example, it may be possible to estimate whether a workpiece can be grasped / pickup using a neural network that receives image data captured by the workpiece imaging device 40 as input. For this purpose, the neural network may be trained in advance and inference may be performed during actual operation. The inference criteria may also be changed by additionally training the neural network on the results of the picking operation history information 300. Alternatively, for a workpiece 202 that has been determined to have been handled in the past by the past handling presence / absence estimation unit 519, the result may be used to determine whether the workpiece can be grasped / pickup.

[0097] Furthermore, if the suction hand 22A and the gripping hand 22B can be changed using a hand changer, the ease of operation may be estimated after changing the suction hand 22A and the gripping hand 22B. In other words, the pick trajectory feasibility estimation unit 534 to the operation time estimation unit 538 perform processing on the suction hand 22A and the gripping hand 22B on the simulator. In this way, the gripping ease estimation unit 535 and the suction ease estimation unit 536 create success / failure estimation information 302 that indicates the execution time and the success / failure of the gripping / suction of each of the picking operation units 2A and 2B. At this time, the workpiece information 3021 and the operation unit data 301 are used as operation unit status information.

[0098] Furthermore, the picking success / failure estimation unit 5 acquires the status of the picking unit 2C where the worker performs the picking work, and estimates the success / failure of the picking unit 2C using the work unit status information indicating this. This includes calculating the execution time based on the worker's ability.

[0099] As described above, the picking success / failure estimation unit 5 estimates the success / failure of each of the picking units 2A to 2C using the workpiece status information of the workpiece 202 to be picked and the work unit status information of the picking units 2A to 2C.

[0100] (Allocation Planning Department 6) The allocation planning unit 6 plans an allocation of which picking unit will perform the picking work of the fixture 201 storing the work to be picked, based on the success / failure estimation information 302 and the allocation information 303. In other words, the allocation planning unit 6 creates the allocation information 303. At this time, it is desirable for the allocation planning unit 6 to use past information, that is, actual results, from the allocation information 303.

[0101] Here, the specific contents of the success / failure estimation information 302 and the allocation information 303 used in this embodiment will be explained. First, FIG. 9 is a diagram showing the success / failure estimation information 302 used in the first embodiment. The success / failure estimation information 302 records the success / failure of the picking work in each picking work unit for each picking target. In FIG. 9, a fixture ID that identifies the fixture 201 is used as the picking target. This fixture ID signifies the unit of picking work. Furthermore, the fixture ID also identifies the work stored in the corresponding fixture. Furthermore, in addition to or in addition to the fixture ID, a work ID that identifies the work 202 may be used.

[0102] In addition, in Figure 9, the success or failure of each process and operation of the picking work, "recognition," "grasping / suction," and "transport operation," as well as the execution time (required time) for these, are used. In Figure 9, the success or failure (success: ○, failure: ×) is recorded for each "recognition," "grasping / suction," and "transport operation." Furthermore, the conditions for success are recorded in parentheses for each of these items. For example, for "recognition" of fixture ID 720 in picking work unit 2A to be successful, it is necessary to use "Algo 1" as the recognition algorithm. Furthermore, the type of hand is recorded in parentheses as a condition for "grasping / suction" to be successful.

[0103] In FIG. 9, the picking operation is recorded in each step of "recognition," "grasping / suction," and "transport operation," but this is not limited to this. For example, the picking operation may be managed as a whole, or "grasping / suction" and "transport operation" may be managed as pick and place. These may also be divided into more detailed steps. Success or failure may also be recorded as a success probability. Some steps, such as execution time, may also be omitted.

[0104] Next, Fig. 10 is a diagram showing allocation information 303 used in the first embodiment. The allocation information 303 is information indicating an operation plan for picking work in the picking system 100. This operation plan may include both past and scheduled data, or may consist of only one of them. Furthermore, the past and scheduled data may be treated as separate information.

[0105] More specifically, the allocation information 303 records the picking target and the status of the picking work for each picking work unit. As shown in FIG. 10, in this embodiment, the picking target is represented by a fixture ID, a work type, and an order quantity. The fixture ID, like the success / failure estimation information 302, signifies the unit of picking work. The work type indicates the work 202 stored in the corresponding fixture ID, i.e., the type of work 202 for which the picking work is performed. It is desirable to use the work shape estimated from the work information 3021 as the work type.

[0106] The number of orders indicates the number of workpieces 202 in a unit of picking work. Here, it is desirable that the number of orders is recorded for each type of workpiece. It is also desirable that these numbers be estimated from the workpiece information 3021.

[0107] In this embodiment, the status, planned completion time, and actual completion time are used as the status of the picking work. Here, the status indicates the progress of the picking work. The planned completion time when the picking work is to be completed and the actual completion time obtained from the picking work history information 300 are also recorded. In this embodiment, the above content is recorded as allocation information 303, but it may also be a database containing, for example, work information 3021, recognizability information 3022, and operability information 3023. Furthermore, the information may be linked to the ID of the fixture and stored in a separate database, but this is not limited to this.

[0108] Next, we will explain the processing of the allocation planning unit 6 using the success / failure estimation information 302 and the allocation information 303. The allocation planning unit 6 first selects a picking work unit to which the work 202 (fixture 201) is to be allocated, using the success / failure estimation information 302. For example, in the case of fixture ID 723 in Figure 9, picking work unit 2B is excluded from the candidates because its item recognition is x.

[0109] The remaining picking work units 2A and 2C are determined by referring to the allocation information 303. For example, the picking work already assigned to the picking work units 2A and 2C may be selected from the one with the earliest scheduled end time. It is also possible to prioritize allocation to a picking work unit that includes a robot to promote automation. Furthermore, the selection criteria may be changed externally, such as by preferentially allocating picking to a worker to speed up the picking work. Furthermore, an interface may be provided for the picking work unit 2C operated by workers, allowing for the appropriate adjustment of work speed and availability, since it is possible for the number of workers to increase or decrease, or for the line to be stopped for breaks, etc.

[0110] Furthermore, in this embodiment, by changing the specifications, allocation to the desired picking unit or a proposal for this allocation can be realized. As an example, consider allocating fixture ID 724 in FIG. 9 to picking unit 2A. First, the allocation planning unit 6 references the hand type for fixture ID 720, which is the previous picking operation performed by picking unit 2A. If the hand types for fixture ID 724 and fixture ID 720 are different, the allocation planning unit 6 outputs a hand change instruction to the work instruction generation unit 7. As a result, the work instruction generation unit 7 outputs an instruction to the worker, etc., to change the hand type using a hand changer or the like when fixture ID 724 arrives. For this purpose, this instruction is output to an output device such as the display screen of picking unit 2A (not shown). The allocation planning unit 6 also updates the information for fixture ID 724 in the allocation information 303. In this case, the completion time of the work can be estimated more accurately by entering the planned completion time that includes the total time required for the hand change and the picking work.

[0111] The allocation planning unit 6 may select a picking unit as follows: First, points indicating the degree of success or failure are assigned to each of "recognition," "grasping / suction," and "transport operation" in the success / failure estimation information 302, and the allocation planning unit 6 makes a judgment based on these points. For example, the allocation planning unit 6 calculates the sum of these points and selects the picking unit with the largest sum.

[0112] Furthermore, if the number of "success (o)"s in the success / failure estimation information 302 is the same, the allocation planning unit 6 selects the picking unit with the shortest execution time. Furthermore, the allocation planning unit 6 may select a picking unit to level the load on the picking units 2A to 2C. This leveling may be performed when the number of "success (o)"s is the same. Furthermore, in the leveling, the criteria may be different between the picking units 2A and 2B equipped with the robots 21A and 21B, the suction hand 22A, and the gripping hand 22B, and the picking unit 2C where a worker works. This enables the above-mentioned preferential allocation (selection) to the robots. Furthermore, in this embodiment, instead of selecting a picking unit, it is desirable to adjust the specifications of the picking units 2A to 2C, such as by replacing the hand 22, or by changing the picking unit. In this way, in this embodiment, the success or failure of the picking operation is estimated according to the status information, and the operation of the picking system including the picking operation unit, particularly the picking operation, is adjusted based on this. More specifically, allocation information 303 indicating an operation plan for the picking operation in the picking system 100 is created.

[0113] With the above configuration, it is possible to estimate in advance whether picking work is possible or not by combining the type and arrangement of hardware such as the line imaging device 23, robot arm 21, and hand 22 on each line with a recognition algorithm for the work 202. Furthermore, by moving the work 202 (furniture 201) to a line with an appropriate picking work section based on the estimated information, or by changing the configuration of the autonomous picking robot on the line, it is possible to prevent a decrease in throughput and damage to goods. [Example]

[0114] In the second embodiment, a line imaging device is used as the workpiece imaging device 40 of the first embodiment. Here, the line imaging device 23B of the picking operation unit 2B is used. That is, the image data of the line imaging device 23B is used to create the success / failure estimation information 302 for each of the picking operation units 2A to 2C.

[0115] FIG. 11 is a diagram for explaining the allocation of picking targets in Example 2. As shown in FIG. 11, this example has a sorting device 8 (e.g., a belt conveyor) that moves workpieces 202 from a picking work unit 2B to other picking work units 2A and 2C. First, the allocation planning unit 6 determines whether picking work is possible at the picking work unit 2B upstream of the sorting device 8. If the allocation planning unit 6 determines that picking work is not possible, it plans picking work at another picking work unit. In this case, the more upstream picking work unit 2A is scheduled with priority over the picking work unit 2C.

[0116] Furthermore, even if the picking work unit 2B determines that picking work is not possible, the allocation planning unit 6 may plan to pick some of the work by the picking work unit 2B and pick other work by the picking work units 2A and 2C.

[0117] The above configuration eliminates the need for additional equipment other than the picking work unit. It also automatically estimates whether each picker can pick the workpiece that has been placed in the fixture that is the target of the picking work. Furthermore, by moving the workpiece to the appropriate picking work unit line based on the estimated information, it is possible to prevent a decrease in throughput and damage to the items. [Example]

[0118] Example 3 targets each picking work group 2 in multiple picking areas. FIG. 12 is a system configuration diagram of a picking system 100 in Example 3. In FIG. 12, a sorting control device 3 in the picking system 100 is connected to a control device 1, which is an information processing device, and a manager terminal group 70 via a network 60. The control device 1 is further connected to picking work groups 2-1 and 2-2 and sorting devices 30-1 and 30-2, which are the objects of control. Here, the picking work group 2-1 and the sorting device 30-1, and the picking work group 2-2 and the sorting device 30-2 are installed at different locations, such as different warehouses. In other words, furniture 201 moves from the sorting device 30-1 to the picking work group 2-1. Also, furniture 201 moves from the sorting device 30-2 to the picking work group 2-2 (see the arrows in the figure). In this way, in this embodiment, control of picking operations at multiple locations is realized by the sorting control device 3. However, the number of locations is arbitrary, and may be a single location.

[0119] The sorting control device 3 is realized by a computer such as a server. Fig. 12 shows its configuration. That is, the sorting control device 3 has a communication device 31, a processing device 32, a memory 33, and a secondary storage device 34, which are connected to each other via a communication path such as a bus. First, the communication device 31 connects to a network 60 and communicates with other devices.

[0120] The processing device 32 can be realized by a processor such as a CPU (Central Processing Unit), and executes processing in accordance with a picking system operation support program 50 stored in a secondary storage device 34 (described later). The picking system operation support program 50 includes a status information acquisition module 501, a picking success / failure estimation module 502, an allocation plan module 503, and a work instruction generation module 504.

[0121] Here, the content of the processing executed based on each of these modules is the same as the processing of each unit shown in Fig. 2. That is, each module has the following corresponding relationship with each unit. Status information acquisition module 501: Status information acquisition unit 4 Picking success / failure estimation module 502: Picking success / failure estimation unit 5 Allocation planning module 503: Allocation planning unit 6 and a work instruction generation module 504: a work instruction generation unit 7 A program may be configured by combining each of these modules or a part of them.

[0122] The memory 33 also stores the picking system operation support program 50 stored in the secondary storage device 34 and information used for processing by the processing device 32. The secondary storage device 34 can be implemented as a so-called storage device, and stores the picking system operation support program 50, picking work history information 300, work unit data 301, success / failure estimation information 302, and allocation information 303. The secondary storage device 34 may also be implemented as various storage media such as an external HDD (Hard Disk Drive), SSD (Solid State Drive), or memory card. Furthermore, the sorting control device 3 may be implemented as a separate device, such as a file server. The memory 33 and secondary storage device 34 correspond to the storage unit described above.

[0123] The manager terminal group 70 is a computer used by a manager who manages the above-mentioned bases, and receives instructions for the sorting control device 3 and outputs processing results. The manager terminal group 70 can be omitted, in which case it is desirable to provide input and output devices in the sorting control device 3. Furthermore, it is not necessary to install multiple manager terminal groups 70, and the sorting control device 3 may be connected without going through the network 60.

[0124] Next, the processing in the above-mentioned sorting control device 3 will be described. Fig. 13 is a flowchart showing the processing in the third embodiment. First, in step S1, the processing device 32 acquires work unit status information of the picking work units 2A to 2C in the picking work unit groups 2-1 and 2-2 in accordance with a status information acquisition module 501 of the picking system operation support program 50. This acquisition may be the same as in the first and second embodiments, or may be created using image data from the line imaging devices 23A and 23B of each picking work unit.

[0125] Furthermore, in step S2, the processing device 32 determines whether the workpiece 202 (furniture 201) to be picked has been placed in each picking area in accordance with the status information acquisition module 501. For this purpose, the imaging data of the workpiece imaging device 40 may be used, or a sensor may be used. That is, a sensor or the like may be used to detect whether the workpiece 202 (furniture 201) has passed a predetermined position, and based on this, it is possible to determine whether the workpiece 202 (furniture 201) has been placed. As a result, if the workpiece 202 has been placed (Yes), the process proceeds to step S3. If the workpiece has not been placed (No), step S2 is repeated.

[0126] In step S3, the processing device 32 acquires workpiece state information about the inserted workpiece 202 in accordance with the state information acquisition module 501. For example, workpiece state information indicating the hardness (softness) of the workpiece 202 as "hard" or "soft" is acquired as the state of the workpiece 202. This allows classification depending on whether the hardness of the inserted workpiece 202 is equal to or greater than a predetermined value.

[0127] Furthermore, in step S4, the processing device 32 generates success / failure estimation information 302 using status information for each picking operation unit of the picking operation unit group to which the workpiece 202 has been introduced, in accordance with the picking success / failure estimation module 502. This step S4 is made up of steps S41 and S42. In step S41, the processing device 32 generates recognition ease information for the picking operation unit group to which the workpiece 202 has been introduced. Also, in step S41, the processing device 32 generates operation ease information for the picking operation unit group to which the workpiece 202 has been introduced. For example, in step S42, the processing device 32 generates operation ease information indicating the ease of operation, in particular the ease of lifting, according to the "hardware" and "software" of the workpiece 202.

[0128] Furthermore, in step S5, the processing device 32 creates allocation information in accordance with the allocation plan module 503. That is, the picking unit that will perform the picking work is identified. For example, if the work status information is "hard," a picking unit that uses a hand with a gripping function is identified. Also, if the work status information is "soft," a picking unit that uses a hand with a suction function is identified. Here, if the work 202 is soft, it is more difficult to grip (grasp) it than to suction it. Therefore, the picking work can be performed by a more appropriate picking unit. This "more appropriate" means that a picking unit that is relatively good at the job is selected from among the picking units.

[0129] In step S6, the processing device 32 creates a work instruction in accordance with the allocation information in accordance with the work instruction generation module 504, and notifies the control device 1 via the communication device 31. This work instruction includes identification information of the selected picking work unit and allocation information 303 of the target workpiece 202 or fixture 201. In other words, the work instruction includes information that identifies the picking work unit and the picking target.

[0130] Then, the control device 1 creates a control signal to move the workpiece 202 to the picking work unit indicated by the work instruction, and notifies the sorting device 30. As a result, the sorting device 30 moves the workpiece 202 (furniture 201) to be picked to the picking work unit indicated by the work instruction.

[0131] The control device 1 also notifies the selected picking work unit of a control command corresponding to the work instruction. More specifically, the control command is notified to the picking work control device 20. As a result, the picking work control device 20 creates a control signal corresponding to the notified control command and controls the hardware of the picking work unit. The notification of this control command may be executed on the condition that the sorting device 30 has moved the work 202 (furniture 201) to the corresponding picking work unit.

[0132] When the picking work section 2C where the worker will work is selected, the control device 1 notifies the terminal device or display device used by the worker of allocation information for the items to be picked. As a result, the terminal device or display device displays the allocation information, thereby assisting the worker in their picking work. The notification of the allocation information to the terminal device or display device may be performed by the sorting control device 3. Furthermore, at least one of the control device 1 and the picking work control device 20 may be omitted. In this case, the sorting control device 3 or the control device 1 controls each device and notifies the information to be displayed. In other words, the sorting control device 3 and the control device 1 perform at least some of the functions of the control device 1 and the picking work control device 20.

[0133] In this embodiment, the processing flow of FIG. 13 is executed each time a workpiece 202 is input, but it may also be executed as a batch process. For example, it is desirable that the sorting control device 3 obtains schedule information for a predetermined period, such as one day, and executes each step of the above flowchart for this information. In this case, step S1 may be executed collectively for the image data of the workpieces 202 to be picked during the relevant period, or workpiece status information for these workpieces 202 may be stored in advance and used. Note that the workpiece status information stored in advance in step S2 may also be used in cases other than batch processing.

[0134] As described above, this embodiment shows an example of implementing the present invention using a so-called cloud system. Therefore, this embodiment makes it possible to efficiently manage picking operations at multiple locations. [Explanation of symbols]

[0135] 1: Control device 2: Picking work group 21: Robot arm 22: Hand 23: Line imaging device 24A: Peripheral installations 25: Lighting 3: Sorting control device 4: Status information acquisition unit 5: Picking success / failure estimation unit 6: Allocation Planning Department 7: Work instruction generation section 20: Picking operation control device 30: Sorting device 40: Workpiece imaging device 50: Picking system operation support program 50 60: Network 70: Administrator terminals 100: Picking system 300: Picking work history information 301: Work unit data 302: Success / failure estimate information 303: Distribution information

Claims

1. A picking system operation support device that supports operations related to picking work in a picking system, a status information acquisition unit that acquires status information indicating the status of a picking target and the status of the picking units that perform picking work on the picking target, the status information acquisition unit being a plurality of picking units included in the picking system; a picking success / failure estimation unit that estimates success or failure of the picking operation in each of the plurality of picking operation units for each of the picking targets using the status information; a distribution planning unit that creates distribution information indicating an operation plan for picking work in the picking system according to the estimated success or failure, the picking success / failure estimation unit creates success / failure estimation information that records the success or failure of each of the plurality of steps of the picking work, the allocation planning unit excludes from candidates any picking operation unit for which any of the plurality of processes indicates a failure in the success / failure estimation information, and selects, using the allocation information, a picking operation unit to perform the picking operation from the plurality of picking operation units that have not been excluded from the candidates; A picking system operation support device that improves operational efficiency regarding picking work in the picking system.

2. 2. The picking system operation support device according to claim 1, The picking success / failure estimation unit is a picking system operation support device that creates, as the success / failure of the picking work, recognition ease information indicating the ease of recognition of the picking work that constitutes the picking work, and operation ease information indicating the ease of lifting and transporting operations.

3. 2. The picking system operation support device according to claim 1, Further, a storage unit is provided to store picking operation history information indicating the results of picking operations in the picking system, The picking success / failure estimation unit further estimates the success / failure of the picking operation using the picking operation history information.

4. 4. The picking system operation support device according to claim 3, The picking success / failure estimation unit is a picking system operation support device that uses the picking operation history information to change the estimation criteria for the success / failure of the picking operation.

5. In the picking system operation support device described in claim 1, the picking success / failure estimation unit creates the success / failure estimation information to which points indicating the degree of success / failure are assigned for each of a plurality of processes of the picking work, The allocation planning unit calculates the sum of the points and selects the picking work unit with the largest sum from the plurality of picking work units.

6. 6. The picking system operation support device according to claim 5, some of the plurality of picking operation units are picking robots, The allocation planning unit is a picking system operation support device that preferentially selects the picking robot.

7. 2. The picking system operation support device according to claim 1, The allocation planning unit is a picking system operation support device that adjusts the specifications of the picking operation unit.

8. A picking system operation support device according to any one of claims 1 to 7; The picking operation unit; A picking system having a control device that outputs a control command to the picking operation unit to perform the picking operation based on the allocation information.

Images