Management system, information processing device, information processing method, and program

Abstract

The system determines whether or not an item can be grasped by a robot. [Solution] The management system of the embodiment includes a robot system including a robot that grasps and picks items, an information processing device that determines whether or not it is possible for the robot to pick the target item, and a controller that instructs the robot to pick the target item that it has been determined is possible for the robot to pick.

Description

Technical Field

[0001] Embodiments of the present invention relate to a management system, an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, in a logistics warehouse or the like, when taking out a specified article from a plurality of stored articles, there is a method of using a robot. The robot is provided with a gripping mechanism such as a suction mechanism or a sandwiching mechanism, for example, and when receiving a control signal, it picks up the specified article by gripping.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, depending on the article, there are some articles that cannot be picked up by a robot for various reasons such as weight, size (which means length, dimensions, etc. as described later), shape, and material. In that case, the operator takes out the article that the robot cannot grip. On the other hand, articles that can be gripped by the robot are registered in a storage unit (DB (Data Base)) on the system in order to be processed by the robot. However, when the number and types of articles are large, there is a problem that manual registration takes time and effort.

[0005] Therefore, the present invention has been made in view of the above circumstances, and aims to provide a management system, an information processing device, an information processing method, and a program that can determine whether or not an item is suitable for picking by a robot. [Means for solving the problem]

[0006] The management system of the embodiment is a management system comprising: a robot system including a robot that grasps and picks articles; an information processing device that determines whether or not the robot can pick an article; and a controller that instructs the robot to pick the article that it has been determined to be pickable. The information processing device includes: an out-of-specification determination unit that determines whether or not the article is outside the specifications for picking based on information belonging to the article; a recognition determination unit that determines whether or not the article can be recognized based on an image of the article; and a picking feasibility determination unit that estimates whether or not the robot can grasp the article and determines whether or not the picking operation is possible based on data related to the robot's grasping operation. The system determines whether or not the robot can pick the article based on the determination results of the out-of-specification determination unit, the recognition determination unit, and the picking feasibility determination unit. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is an overall diagram of the item management system according to the first embodiment. [Figure 2] Figure 2 is a flowchart showing the information processing for item registration by the item management system of the first embodiment. [Figure 3] Figure 3 shows an example of data from the first embodiment. [Figure 4] Figure 4 shows an example of data from the second embodiment. [Figure 5] Figure 5 shows an example of data from the third embodiment. [Figure 6]Figure 6 shows an example of data from the fourth embodiment. [Figure 7] Figure 7 is a flowchart showing the processing performed by the item management system of the fifth embodiment. [Figure 8] Figure 8 shows an example of data from the sixth embodiment. [Modes for carrying out the invention]

[0008] The following describes in detail embodiments (first to sixth embodiments) and modifications of the management system, information processing device, information processing method, and program of the present invention with reference to the attached drawings. In the description of the second embodiment and subsequent embodiments, explanations of matters similar to those described previously will be omitted as appropriate.

[0009] Furthermore, in the following embodiments, robotic picking operations may be simply described as "picking" or "picking process," as having an equivalent meaning.

[0010] Figure 1 is an overall configuration diagram of the first embodiment of the item management system S (management system). The item management system S (management system) is used, for example, for item management in a logistics warehouse and comprises a higher-level system 1, an information processing device 2, a controller 3, and a robot system 4. The information processing device 2 and the controller 3 constitute an information processing device that registers items to be picked by the robot 43 from among a plurality of items in the item DB 311 (registration unit) of the storage unit 31.

[0011] The higher-level system 1 is a computer system and includes a master database 11. The master database 11 stores information for each of multiple items, such as item ID (Identifier), item name, weight, size, shape, material, and captured image. Here, "size" refers to, for example, the length and dimensions in the vertical, horizontal, and depth directions. A detailed explanation of "size" will be omitted as appropriate below.

[0012] The information processing device 2 is a computer device, and includes a storage unit 21, an input unit 22, an output unit 23, a communication unit 24, and a control unit 25.

[0013] The storage unit 21 is a storage device such as a HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various information. The storage unit 21 stores, for example, various data, various calculation results, and the like.

[0014] The input unit 22 is an input device that receives a user's operation on the information processing device 2, and is, for example, a keyboard, a mouse, a touch panel, or the like.

[0015] The output unit 23 is a means for outputting various information, and is, for example, a display device such as a liquid crystal display device (LCD (Liquid Crystal Display)), or an audio output device such as a speaker.

[0016] The communication unit 24 is a communication interface for communicating with other devices such as the upper system 1 and the controller 3. Further, the communication unit 24 is configured to be able to communicate with devices, systems, etc. outside the article management system S.

[0017] The control unit 25 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The CPU comprehensively controls the operation of the information processing device 2. The ROM is a storage medium that stores various programs and various data. The RAM is a storage medium for temporarily storing various programs and rewriting various data. Then, the CPU executes the programs stored in the ROM, the storage unit 21, etc. using the RAM as a work area (working area).

[0018] The control unit 25 executes various controls. As a functional configuration, the control unit 25 includes an acquisition unit 251, a first out-of-specification determination unit 252, a second out-of-specification determination unit 253, a recognition determination unit 254, a determination unit 255 for whether actual machine picking is possible, a determination unit 256 for whether picking is possible, and a processing unit 257.

[0019] The acquisition unit 251 acquires various information from the upper-level system 1 or the controller 3. Note that this various information can also be acquired from outside the article management system S via the communication unit 24 or the like.

[0020] The first out-of-specification determination unit 252 determines whether it is out of specification in picking with respect to an article for which it is determined whether picking is possible (hereinafter, also referred to as "article to be determined"). Based on at least the size information and weight information of the article, this determination is made. These size information and weight information can be acquired from the upper-level system 1 via the acquisition unit 251. It is also possible to acquire them from outside the article management system S via the communication unit 24 or the like.

[0021] The second out-of-specification determination unit 253 determines that it is out of specification in the picking operation of the robot when the article meets the conditions arbitrarily set regarding being out of specification. For example, when there is data indicating that it is out of specification in the picking operation with respect to the article to be determined, the second out-of-specification determination unit 253 determines that it is out of specification. Here, conditions are set in advance, and when the article to be determined has data corresponding to the conditions, it is determined that the article is out of specification in the picking operation.

[0022] For example, items that are already broken, and items that are likely to break during the picking process (e.g., glass products, ceramics, etc.) can be classified as non-specified items in the picking process. If an item being evaluated meets this condition, it is classified as non-specified in the picking process. Such items can be judged as non-specified based on their appearance, etc. Similarly, items whose lids may open during the gripping and transporting operations of a robot during picking can also be classified as non-specified items. Such items can also be judged as non-specified from the perspective of their composition or function. In the above, any conditions can be set to determine whether an item is non-specified in the picking process.

[0023] Furthermore, the first non-specification determination unit 252 and the second non-specification determination unit 253 can be integrated into a single functional configuration (corresponding to a single non-specification determination unit). Regarding the feasibility of picking an item, the determination of whether or not it is non-specification, based on at least the size and weight information of the item as described above, and the determination of whether or not it is non-specification in terms of picking, can be performed on a single functional configuration.

[0024] The recognition determination unit 254 determines whether an item to be determined is recognizable or not based on the captured image of the item. For example, the recognition determination unit 254 rotates, enlarges, reduces, or adds noise information to the captured image of the item, and if the recognition area of ​​the item in the image does not change, it determines that the item is recognizable. If the recognition area of ​​the item in the image changes, it determines that the item is unrecognizable. However, the recognition determination method is not limited to this.

[0025] The machine-less picking feasibility determination unit 255 estimates whether the robot 43 can grasp the item to be determined, and determines whether the picking operation is feasible based on the result of this estimation. Here, the machine estimates whether the robot 43 can grasp the item to be determined, and determines whether the picking operation is feasible, based on the picking process data obtained without performing motion verification using the robot 43 (hereinafter referred to as machine-less data). Specifically, the picking process data obtained without performing motion verification is data related to the grasping operation of the robot 43. In this way, without actually performing motion verification using the robot, the machine estimates whether the item to be determined can be grasped, and determines whether the picking operation is feasible based on the result.

[0026] For example, if the robot system 4 has multiple types of robot hands 43, the actual picking feasibility determination unit 255 estimates whether the robot 43 can pick an item based on the picking feasibility information for each item in each hand. Then, based on the result, it determines whether the picking operation for the item to be determined is feasible. (Details will be described later using Figure 3.)

[0027] As data without physical equipment, for example, existing data (such as processing performance data) obtained from a higher-level system or from outside the item management system S can be used by the information processing device 2. Specifically, in the item management system S shown in Figure 1, the information processing device 2 can use existing data such as accumulated data obtained from the master DB 11 of the higher-level system 1. In addition, the physical-less picking feasibility determination unit 255 can use separately prepared simulation results to estimate whether or not it is possible to grasp the item to be determined, and then determine whether or not the picking operation is feasible based on that result. For example, a robot system can be constructed in a virtual world, and based on the captured images, an operation simulation can be performed to determine whether or not the robot is able to grasp the item to be determined. Then, the feasibility of the picking operation can be determined using the results.

[0028] The picking feasibility determination unit 256 comprehensively determines whether the robot 43 can pick the item based on the necessary determination results from the first non-specification determination unit 252, the second non-specification determination unit 253, the recognition determination unit 254, the machine-less picking feasibility determination unit 255, and the machine-picking feasibility determination unit 321 (details described later). If it determines that the item can be picked, it registers the item in the item DB 311 of the storage unit 31.

[0029] Furthermore, it is possible to integrate the recognition determination unit 254 and the actual machine-less picking feasibility determination unit 255 into a single functional configuration (a single determination unit). In such a configuration, it is possible to estimate whether or not the item to be judged can be grasped based on the recognition results using AI (Artificial Intelligence) or machine learning, and to determine whether or not the picking operation is feasible based on the result.

[0030] The processing unit 257 performs various calculation processes. For example, when at least some of the functions constituting the item management system S are updated, the processing unit 257 determines which of the first non-specification determination unit 252, the second non-specification determination unit 253, the recognition determination unit 254, the actual machine-less picking feasibility determination unit 255, and the actual machine picking feasibility determination unit 321 require re-determination, and causes only those that require re-determination to perform re-determination.

[0031] Controller 3 is a computer device that instructs the robot 43 of the robot system 4 to pick items from among multiple items that are registered in the item DB 311 of the storage unit 31. Controller 3 comprises a storage unit 31 and a control unit 32.

[0032] The storage unit 31 stores various types of information. The storage unit 31 includes an item database 311 and an operation history database 312.

[0033] The item database 311 is a database that registers the items that will be picked by the robot 43 from among a group of items.

[0034] The operation history DB312 is a database that stores the operation history of the robot 43.

[0035] The control unit 32 performs various controls. The control unit 32 includes a robot picking feasibility determination unit 321. The robot picking feasibility determination unit 321 determines whether or not the robot can perform picking work with respect to the item to be determined, based on the results of operation verification using the robot 43.

[0036] Robot system 4 is a system for picking items and includes a camera 41, a sensor 42, and a robot 43.

[0037] Camera 41, for example, consists of a color camera and a three-dimensional camera, and photographs items inside the item container and outputs the captured images, as well as measuring the distance to each position of the items and outputting distance information.

[0038] Sensor 42 includes various types of sensors. Sensor 42 includes, for example, a sensor installed near camera 41 that measures a predetermined physical quantity related to an item grasped (picked) by robot 43.

[0039] The robot 43 is a device that picks up items, for example, by a suction mechanism, in response to a control signal from the controller 3.

[0040] In addition to the camera 41, sensor 42, and robot 43, the robot system 4 also includes, for example, a power supply unit for operating various drive units, a cylinder for storing compressed air, a compressor, a vacuum pump, an external interface such as a UI (User Interface), and safety mechanisms such as a light curtain and a collision detector.

[0041] Next, an example of information processing for registering items using the item management system S will be described with reference to Figures 2 and 3. Figure 2 is a flowchart showing the information processing for registering items using the item management system S of the first embodiment. Figure 3 is a diagram showing an example of data of the first embodiment. Figure 3(a) is an example of data for each item. Figure 3(b) is data showing whether each of the multiple hands of the robot 43 can grasp each item. In the following example, the robot 43 is equipped with hand 1 and hand 2, but not hand 3.

[0042] In step S1, the acquisition unit 251 of the control unit 25 in the information processing device 2 acquires information about the new item (for example, information such as item ID, item name, weight, size, shape, material, and captured image) from the master DB 11 of the higher-level system 1. Furthermore, information on these new items can also be obtained from sources other than the item management system S, such as the communication unit 24.

[0043] Next, in step S2, the first non-specification determination unit 252 performs a first non-specification determination with respect to the item, determining whether it is non-specific for picking operations based on at least the item's size information and weight information. Based on the result, it assigns a flag (size / weight non-specification flag in Figure 3(a)).

[0044] Next, in step S3, the second out-of-specification determination unit 253 performs a second out-of-specification determination if there is data indicating that the item is out of specification for the picking operation. Here, predetermined conditions are set, and if there is an item that meets those conditions, it is assumed that the item has data indicating that it is out of specification and cannot be detected in the picking operation. For example, items that are already broken, items that may break during the picking operation (e.g., glass products, ceramics such as pottery, etc.), and items whose lids may open during the gripping and transporting operations of the robot picking operation are treated as out-of-specification items. Based on the result, a flag (undetectable out-of-specification flag in Figure 3(a)) is assigned.

[0045] Furthermore, steps S2 and S3 can also be performed on a single functional configuration. As mentioned above, the first non-specification determination unit 252 and the second non-specification determination unit 253 can be integrated into a single functional configuration (a single non-specification determination unit). Therefore, steps 2 and S3 can be performed as a single processing step on a single functional configuration.

[0046] Next, in step S4, the recognition determination unit 254 performs a recognition determination with respect to the item, determining whether or not the item is recognizable based on the captured image of the item. Then, based on the determination result, it assigns a flag (recognition feasibility flag in Figure 3(a)).

[0047] Next, in step S5, the machine-less picking feasibility determination unit 255 estimates whether the robot 43 can grasp the item to be determined based on the picking process data obtained without performing operational verification using the robot 43 (i.e., without actually performing operational verification using the robot 43). Then, based on the result of this estimation, it determines whether the item to be determined can be picked. In this way, the feasibility of picking the item to be determined is determined without actually performing operational verification using the robot, and a flag (grasp feasibility flag in Figure 3(a)) is assigned based on the determination result.

[0048] Specifically, for example, the actual machine picking feasibility determination unit 255 determines whether a picking operation is possible based on the gripping feasibility information for each item for each hand in Figure 3(b). In Figure 3(b), only item B, which is marked "×" (not gripping) for both hand 1 and hand 2, will have a picking feasibility determination result (gripping feasibility flag) of "×" (not).

[0049] Picking feasibility information for each hand can be obtained, for example, from simulation results or existing data based on robot operation results. Picking feasibility information for each hand can be obtained from the higher-level system 1 within the item management system S in Figure 1. Picking feasibility information for each hand can also be obtained from a system outside of the item management system S in Figure 1. From the external system, it can be obtained via the communication unit 24 within the information processing device 2 in Figure 1.

[0050] Here, instead of the picking feasibility determination result for each hand, you may use the feasibility determination result for each robot type (e.g., robot model A, robot model B) as "○" (grasp "possible") or "×" (grasp "impossible"). Alternatively, you may use both the data for each hand and the picking / grasp feasibility determination result for each robot type.

[0051] Furthermore, steps S2 and S3 can also be performed on a single functional configuration. As mentioned above, the first non-specification determination unit 252 and the second non-specification determination unit 253 can be integrated into a single functional configuration (a single non-specification determination unit). Therefore, steps 2 and S3 can be performed as a single processing step on a single functional configuration.

[0052] Furthermore, by integrating the functions of the recognition determination unit 254 in step S4 and the actual machine-less picking feasibility determination unit 255 in step S5, picking feasibility can also be determined based on recognition results using AI or machine learning.

[0053] In this embodiment, the determinations in steps S2, S4, and S5 of Figure 2 are performed, and if an item is flagged as ○ (OK) in at least all of these steps, it can be determined in step S7, described later, that the item can actually be picked by the robot. Furthermore, the determination result can be registered in the item DB311, also described later. On the other hand, if the flag is × (NG) in any one of the steps S2, S4, and S5, it can be determined that the item cannot be picked by the robot.

[0054] Furthermore, in addition to the determinations in steps S2, S4, and S5, step S6 in Figure 2 allows for verification by actually operating the robot, enabling a more accurate determination of whether or not an item can be picked.

[0055] In other words, in this embodiment, for example, not all of steps S2 to S6 in Figure 2 are necessary, and some steps may be omitted, except for steps S2, S4, and S5.

[0056] In step S6, the actual picking feasibility determination unit 321 in the control unit 32 of the controller 3 performs an actual picking feasibility determination for the item to be determined, based on the results of the operation verification using the robot 43, to determine whether or not the robot 43 can perform the picking operation. Then, based on the determination result, a flag (actual machine verification feasibility flag in Figure 3(a)) is assigned.

[0057] Next, in step S7, the picking feasibility determination unit 256 comprehensively determines whether the robot 43 can pick the item based on the determination results (each assigned flag) from steps S2, S4, and S5. Based on this determination, the picking feasibility determination unit 256 assigns a robot handling feasibility flag of ○ (yes) or × (no) (shown in Figure 3(a)) to the item, indicating whether it can be handled by the robot. That is, items for which the flag is ○ (yes) in at least all of the determination results from steps S2, S4, and S5 are determined to be items that can actually be picked by the robot. On the other hand, items for which the flag is × (no) in any one of the determination results from steps S2, S4, and S5 are determined to be items that cannot actually be picked by the robot.

[0058] Furthermore, in step S7, the picking feasibility determination unit 256 may comprehensively determine whether picking by the robot 43 is possible based on the determination results (each assigned flag) from steps S2 to S6, and assign a robot handling feasibility flag (Figure 3(a)) according to the determination result. In that case, as with item A shown in Figure 3(a), if all five flags corresponding to the determination results from steps S2 to S6 are ○ (possible), the robot handling feasibility flag is set to ○ (possible). On the other hand, items B and C shown in Figure 3(a) each have × (not possible) in one of the five flags corresponding to the determination results from steps S2 to S6. For item B, the determination result of step S5 is × (not possible to grasp) among steps S2, S4, and S5. For item C, the determination result of step S4 is × (not possible to grasp) among steps S2, S4, and S5. Therefore, the robot handling feasibility flag for items B and C is × (not possible to grasp).

[0059] Next, in step S8, the picking feasibility determination unit 256 determines whether the robot handling feasibility flag (Figure 3(a)) is feasible (○) or not (×). If it is Yes, the process proceeds to step S9; otherwise, the process ends.

[0060] In step S9, the picking feasibility determination unit 256 registers the item in the item DB 311 of the storage unit 31. In other words, as shown in Figure 3(a), only item A, for which the robot handling feasibility flag is ○ (acceptable), is registered in the item DB 311.

[0061] Thus, according to the item management system S of the first embodiment, it is possible to determine whether or not an item can be picked by the robot 43 by performing the above-described judgments and flag assignments. Therefore, the time and effort required for product registration can be reduced.

[0062] In this embodiment, the determinations in steps S2, S4, and S5 of Figure 2 are performed, and items for which the flag is ○ (OK) in at least all of these steps are determined to be actually pickable and can be registered in the item DB 311.

[0063] Furthermore, by adding other necessary steps, as shown in step S6 of Figure 2, in addition to the determinations in steps S2, S4, and S5, it is possible to determine whether an item can be picked with a higher degree of accuracy.

[0064] In subsequent embodiments, the system configuration, processing flow, and example data used are basically the same as those described in the first embodiment. Furthermore, any changes are specifically described below.

[0065] (Second Embodiment) Next, a second embodiment will be described. Figure 4 shows an example of data in the second embodiment. Figure 4(a) is an example of data for each item. Figure 4(b) shows data indicating the payload capacity of each of the multiple hands of the robot 43. In step S2 of Figure 2, the first non-specification determination unit 252 performs a first non-specification determination for the item, using the data in Figure 3(b) in addition to the size information and weight information.

[0066] In that case, if the payload capacity requirement of at least one of hands 1 and 2 is met, the size / weight specification non-specification flag is set to ○ (OK). From Figures 4(a) and (b), items A to C meet the payload capacity requirement of either hand 1 or 2, so the weight specification non-specification flag is set to ○ (OK). On the other hand, item D has a weight of 5000 (g) and does not meet the payload capacity requirement of both hands 1 and 2, so the size / weight specification non-specification flag is set to × (NG). In steps S2, S4 and S5, the judgment result of step S2 is × (NG). Then, for item D, regardless of the judgment results of steps S3 to S6 in Figure 2, it is confirmed that × (NG) will be assigned as the robot handling feasibility flag in step S7. Therefore, for item D, the processing of steps S3 to S6 can be skipped, and × (NG) can be assigned as the robot handling feasibility flag in step S7.

[0067] In the second embodiment, the feasibility of picking the items to be judged by the robot 43 is determined as described above. Items that are determined to be pickable by the robot 43 are registered in the item DB 311 of the storage unit 31.

[0068] Thus, according to the second embodiment, processing can be sped up by skipping unnecessary processing.

[0069] (Third embodiment) Next, a third embodiment will be described. Figure 5 shows an example of data from the third embodiment. The robot system 4 is already equipped with hands 1 and 2 as shown in Figure 5(b), and a new hand 3 is added. With either hand 1 or 2, items A and C are determined to have a gripping flag of ○ (possible). With item B, both hands 1 and 2 are determined to have a gripping flag of × (not possible). Therefore, with the newly added hand 3, we consider whether item B can be gripped among items A, B, and C.

[0070] In that case, the processing unit 257 of the control unit 25 of the information processing device 2 identifies the step among steps S2 to S6 in Figure 2 that requires re-evaluation. Since the addition of the new hand 3 does not affect the evaluation results of steps S2 to S4, steps S2 to S4 are skipped, and it is determined that re-evaluation of steps S5 and S6 is required. In other words, in this case, if only the hand has been changed, it is sufficient to evaluate the elements related to the gripping of the item by the hand, and evaluation in the other steps does not need to be performed.

[0071] Then, during the re-evaluation, in step S5, the robot-less picking feasibility determination unit 255 determines whether or not the robot can perform the picking operation with respect to item B, based on the picking feasibility information for each hand in Figure 5(b). In Figure 5(b), since item B is marked as ○ (graspable) with hand 3, the gripping feasibility flag is changed from × (not grippable) to ○ (graspable). In other words, when hand 3 is used, the determination result for item B will be ○ (graspable) in all of steps S2, S4, and S5, and in step S7, it can be determined that the robot can perform the picking operation.

[0072] Next, in step S6, the actual picking feasibility determination unit 321 performs a picking feasibility determination for item B based on the results of the operation verification using the robot 43, and assigns an actual machine verification feasibility flag. Here, the actual machine verification feasibility flag is changed from × (not possible) to ○ (possible).

[0073] Next, in step S7, the picking feasibility determination unit 256 determines that all five flags from steps S2 to S6 for item B are ○ (feasible), so it changes the robot handling feasibility flag from × (not possible) to ○ (feasible).

[0074] Thus, according to the third embodiment, when a hand is added, only the steps S2 to S6 in Figure 2 that require re-evaluation are re-evaluated. This makes it possible to register items in response to the addition of a hand with minimal processing.

[0075] In the third embodiment, the feasibility of picking the items to be judged by the robot 43 is determined as described above. Items that are determined to be pickable by the robot 43 are registered in the item DB 311 of the storage unit 31.

[0076] (Fourth Embodiment) Next, a fourth embodiment will be described. Figure 6 shows an example of data according to the fourth embodiment. If the item to be judged is similar to an item that has already been judged, the judgment by at least one of the second non-specification determination unit 253 and the recognition determination unit 254 is skipped.

[0077] In the example in Figure 6, item C' is assumed to be similar to item C in appearance, such as shape, contour, and surface condition, but different in size and weight. In this case, for example, for item C', both the determination in step S3 by the second non-specification determination unit 253 and the determination in step S4 by the recognition determination unit 254 can be skipped, and the determination results of steps S3 and S4 already determined for item C can be copied and reused.

[0078] In the fourth embodiment, as shown in Figure 6, item C' will have a robot handling feasibility flag of × (not) similar to item C, and will be determined not to be pickable by a robot.

[0079] Thus, according to the fourth embodiment, if the item to be judged has elements similar to an item that has already been judged, some of the judgments are skipped and the judgment results are copied and reused. This makes it easier to register items similar to items that have already been judged.

[0080] In the fourth embodiment, the feasibility of picking the items to be judged by the robot 43 is determined as described above. Items that are determined to be pickable by the robot 43 are registered in the item DB 311 of the storage unit 31.

[0081] (Fifth embodiment) Next, a fifth embodiment will be described. The actual picking feasibility determination unit 321 of the control unit 32 of the controller 3 can perform the actual picking feasibility determination in verification mode as well as normal mode.

[0082] This will be explained with reference to Figure 7. Verification modes can be implemented, for example, in a picking robot, by a safety mode that verifies the operation at a slower speed than the normal mode, and an operation mode that verifies the picking operation state. Examples of operation modes include, in the case of a hand that has picked up an item, extending the time for determining the vacuum level of the suction part, or shaking the hand while it is holding an item to check whether the item falls off.

[0083] The verification mode can be performed using either the safety mode or the operating mode described above, as needed.

[0084] Figure 7 is a flowchart showing the processing performed by the item management system S of the fifth embodiment. By using this processing flowchart, it is possible to determine whether or not robotic picking is possible for the item to be evaluated. Furthermore, the fifth embodiment will be explained with reference to the previously described information regarding the configuration of the item management system in Figure 1 and the flowchart showing the information processing for item registration in Figure 2.

[0085] Here, we assume there are three types of gripping feasibility flags (see Figure 3(a), etc.): "Yes", "Provisional", and "No". An item with a gripping feasibility flag of "Provisional" is, for example, one in Figure 2 where the flag becomes ○ in step S2. An item with a gripping feasibility flag of "Provisional" also includes items where the flag becomes ○ in step S2 of Figure 2, and again in step S3 of the same Figure 2. As shown in the processing flowchart of Figure 7, items with a gripping feasibility flag of "Provisional" will undergo feasibility determination in steps other than steps 2 and S3 in Figure 2.

[0086] In step S11, the processing unit 257 of the control unit 25 of the information processing device 2 determines whether the gripping capability flag is "possible", "temporary", or "not possible". If it is "possible", proceed to step S25; if it is "temporary", proceed to step S12; and if it is "not possible", proceed to step S22.

[0087] If the gripping feasibility flag for an item is already set to "No", in step S22, the processing unit 257 performs a recovery operation with rejection (pickup failed by robot 43) and terminates the process (=end).

[0088] If the gripping feasibility flag for an item is already set to "possible," steps S25 to S28 perform a determination of whether or not the item can be picked up in normal mode. In step S25, the actual picking feasibility determination unit 321 performs an actual machine operation verification in normal mode.

[0089] Next, in step S26, the actual machine picking feasibility determination unit 321 registers the actual machine operation verification result in the storage unit 31.

[0090] Next, in step S27, the actual machine picking feasibility determination unit 321 determines whether or not it is necessary to modify the actual machine verification feasibility flag. If Yes, the process proceeds to step S28; otherwise, the process in the processing flowchart ends (=end).

[0091] In step S28, the actual machine picking feasibility determination unit 321 assigns "No" (×) to the actual machine verification feasibility flag, and terminates the processing in the processing flowchart (=end).

[0092] If the gripping feasibility flag for an item is still "provisional", in step S12, the processing unit 257 takes a photograph of the item using the camera 41 or the like, and the acquisition unit 251 acquires the captured image.

[0093] Next, in step S13, the recognition determination unit 254 performs a recognition determination based on the captured image of the item to determine whether or not the item is recognizable, and assigns a recognition feasibility flag. Alternatively, the recognition processing result and the recognition determination result at this time may be displayed on the GUI (Graphical User Interface) of the output unit 23, and if the user inputs OK via the input unit 22, the system may proceed to the next step S14.

[0094] Next, in step S14, the processing unit 257 determines whether the recognition feasibility flag is "Yes" (○) or "No" (×). If it is Yes ("Yes" (○). This will be omitted from now on.), it proceeds to step S15. If it is No ("No" (×). This will be omitted from now on.), it proceeds to step S23.

[0095] In step S23, the processing unit 257 performs a recovery operation upon rejection (robot 43 fails to pick the item).

[0096] Next, in step S24, the actual machine picking feasibility determination unit 321 assigns "No" (×) to the actual machine verification feasibility flag, and terminates the processing in the processing flowchart (=end).

[0097] In step S15, the machine-less picking feasibility determination unit 255 performs a picking feasibility determination by the robot 43 without using the actual machine and assigns a gripping feasibility flag. Here, as described above, the feasibility of gripping by the robot 43 is estimated based on the picking process data obtained without performing operational verification using the robot 43 (i.e., without actually performing operational verification using the actual machine). Then, based on the result of this estimation, it is determined whether or not the robot 43 can perform the picking operation, and a gripping feasibility flag is assigned to the target item based on the determination result. Alternatively, the determination result at this time may be displayed on the GUI of the output unit 23, and if the user inputs OK via the input unit 22, the process may proceed to the next step S16.

[0098] Next, in step S16, the processing unit 257 determines whether the gripping capability flag is "possible" (○) or not. If it is Yes, the unit proceeds to step S17; if it is No ("not possible" (×)), the unit proceeds to step S23.

[0099] In step S17, the actual machine picking feasibility determination unit 321 performs actual machine operation verification in verification mode. Here, the verification mode can be performed as described above.

[0100] Next, in step S18, the actual machine picking feasibility determination unit 321 registers the actual machine operation verification result in the storage unit 31.

[0101] Next, in step S19, the actual machine picking feasibility determination unit 321 determines whether the actual machine gripping was successful or not. If Yes, the unit proceeds to step S20; otherwise, it proceeds to step S24.

[0102] In step S20, the actual machine picking feasibility determination unit 321 determines whether or not the specified number of actual machine verification operations have been performed. If the result is Yes, the process proceeds to step S21; otherwise, the process ends.

[0103] In step S21, the actual machine picking feasibility determination unit 321 assigns "Fulfillable" (〇) to the actual machine verification feasibility flag, and terminates the processing in the processing flowchart (=end).

[0104] Thus, according to the fifth embodiment, if the gripping feasibility flag is "provisional", in step S12 and beyond, the picking feasibility is determined using the actual robot in a safety mode (verification mode) in which the robot's movements are slower than in the normal mode. This reduces the possibility of the item being damaged during the actual robot operation verification.

[0105] In the fifth embodiment, the feasibility of picking the items to be judged by the robot 43 is determined as described above. Items that are determined to be pickable by the robot 43 are registered in the item DB 311 of the storage unit 31.

[0106] (Sixth Embodiment) Next, a sixth embodiment will be described. Figure 8 shows an example of data according to the sixth embodiment. In the item management system S, when updating the functions related to the recognition determination unit 254 and performing a re-determination for registering an item in the item DB 311 of the storage unit 31, the determinations of the first out-of-specification determination unit 252 and the second out-of-specification determination unit 253 are skipped. In the following, we will focus on item C in Figure 8.

[0107] As an example, the recognition determination unit 254 has been updated to improve its recognition function for target items.

[0108] Specifically, as shown in Figure 8, for item C, the process for assigning the size / weight non-specification flag (step S2 in Figure 2) and the process for assigning the undetectable non-specification flag (step S3 in Figure 2) are skipped, and both flags remain as ○ (acceptable: not non-specification). These processes (steps S2 and S3) are not affected by the update of the recognition determination unit 254, so they can be skipped.

[0109] Then, in step S4 of Figure 2, the recognition determination unit 254 performs a recognition determination based on the captured image of item C to determine whether or not item C is recognizable, and assigns a recognition feasibility flag. Here, as shown in Figure 8, the recognition feasibility flag changes from × (not) to ○ (recognizable).

[0110] Subsequently, a determination was made again in step S5 of Figure 2. Here, in step S5, as described above, the feasibility of gripping by the robot 43 is estimated based on the picking process data obtained without performing operational verification using the robot 43 (i.e., without actually performing operational verification with a real machine).

[0111] As shown in Figure 8, the re-determination result in step S5 remained unchanged from ○ (graspable). Thus, the determination result in all of steps S2, S4, and S5 is ○ (graspable), and in step S7 of Figure 2, the picking feasibility determination unit 256 can determine that item C can be picked by the robot. In addition, in step S7 of Figure 2, the picking feasibility determination unit 256 changes the robot handling feasibility flag for item C from × (not possible) to ○ (possible).

[0112] Furthermore, in step S6 of Figure 2, a second determination can be made. As shown in Figure 8, in the second determination in step S6, a result of ○ (graspable) was obtained based on the operation verification by the robot 43. Thus, the determination results in steps S5 and S6 remain unchanged from ○ (graspable). In that case, in step S7, as shown in Figure 8, the picking feasibility determination unit 256 changes the robot handling feasibility flag from × (not possible) to ○ (possible) because all five flags from steps S2 to S6 are ○ (possible). By performing step S6 in Figure 2, it becomes possible to add the determination result based on the actual operation verification result by the robot in addition to the re-determination result based on step S5.

[0113] Since steps S5 and S6 are affected by the update of the functions related to the recognition determination unit 254 mentioned above, a re-determination of the target item C is performed.

[0114] Thus, according to the sixth embodiment, when a function related to the recognition determination unit 254 is updated, only the step S2 to S6 in Figure 2 that requires re-determination is re-determined. This makes it possible to register items in response to updates to the functions related to the recognition determination unit 254 with minimal processing.

[0115] In the sixth embodiment, the feasibility of picking the items to be judged by the robot 43 is determined as described above. Items that are determined to be pickable by the robot 43 are registered in the item DB 311 of the storage unit 31.

[0116] (modified version) In Figure 5(b), the items that can be grasped differ for each hand, and a graspability flag is determined accordingly. Similarly, there could be variations for each algorithm or sensor, and the graspability flag could be determined accordingly.

[0117] Furthermore, in step S5 of Figure 2, the gripping feasibility flag may be determined using the captured image of the item. Specifically, for example, if the captured image of the item is analyzed and the surface of the item is uneven, or if there is no flat area on the surface near the center of gravity of the item, the gripping feasibility flag may be set to "×" (not possible) because the robot 43's suction mechanism cannot pick it up. While such a determination can be made using AI, for example, it is not limited to this, and other methods such as pattern matching may also be used.

[0118] In addition to the item ID, item name, weight, size, shape, material, and photographed image, the following information may also be used as information about the item to be judged. This information can be used, for example, in steps S2, S3, S5, and S6 in Figure 2. The information regarding whether or not the surface of the item to be judged is indented when pressed can be used in step S4 for recognition feasibility determination, step S5 for picking feasibility determination, and step S6 for actual machine picking feasibility determination, which involves actual machine verification. The information regarding whether the surface of the item to be judged is adsorbable can be used in step S2 for determining whether it is outside of specifications, step S5 for determining whether it can be picked, and step S6 for determining whether it can be picked on the actual machine for verification. In addition, information such as whether the item to be judged is fragile, or whether the item's lid may open during the gripping and transporting operations of a robot during picking, can be used in the out-of-specification determination step S3 of the detection-undetectable out-of-specification determination.

[0119] Furthermore, the programs executed by the information processing device 2 and controller 3 of this embodiment can be provided as installable or executable files recorded on a computer-readable recording medium such as a CD (Compact Disc)-ROM (Read Only Memory), flexible disk (FD), CD-R (Recordable), or DVD (Digital Versatile Disk). Alternatively, these programs may be provided or distributed via a network such as the Internet.

[0120] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0121] For example, the picking mechanism of robot 43 is not limited to a suction mechanism, but may also be other mechanisms such as a gripping mechanism. [Explanation of symbols]

[0122] 1...Host System, 2...Information Processing Unit, 3...Controller, 4...Robot System, 11...Master DB, 21...Storage Unit, 22...Input Unit, 23...Output Unit, 24...Communication Unit, 25...Control Unit, 31...Storage Unit, 32...Control Unit, 41...Camera, 42...Sensor, 43...Robot, 251...Acquisition Unit, 252...First Out-of-Specification Determination Unit, 253...Second Out-of-Specification Determination Unit, 254...Recognition Determination Unit, 255...Actual Machine Picking Feasibility Determination Unit, 256...Picking Feasibility Determination Unit, 257...Processing Unit, 311...Item DB, 312...Operation History DB, 321...Actual Machine Picking Feasibility Determination Unit, S...Item Management System

Claims

1. A management system comprising: a robot system including a robot that grasps and picks items; an information processing device that determines whether or not the robot can pick the target item; and a controller that instructs the robot to pick the target item that it has been determined to be pickable by the robot, The aforementioned information processing device is A non-specification determination unit determines whether the target item is outside the specifications for the picking operation based on information relating to the target item, A recognition determination unit that determines whether or not the item can be recognized based on an image of the item, The system includes a picking feasibility determination unit that estimates whether the robot can grasp the target item based on data related to the robot's gripping operation, and determines whether the picking operation is feasible. A management system that determines whether or not the robot can perform the picking operation on the target item based on the determination results of the non-specification determination unit, the recognition determination unit, and the picking feasibility determination unit.

2. The management system according to claim 1, wherein the information processing device determines that the target item can be picked by the robot if the determination result in the non-specification determination unit is not non-specification, the determination result in the recognition determination unit is recognizable, and the determination result in the pickability determination unit is that the item can be grasped.

3. The management system according to claim 2, wherein the information processing device performs the following determinations in the order of determination by the non-specification determination unit, determination by the recognition determination unit, and determination by the picking feasibility determination unit.

4. The management system according to claim 1, further comprising a registration unit for storing information of items that the robot has determined can be picked.

5. The management system according to claim 1, wherein the out-of-spec determination unit has a first out-of-spec determination unit that determines whether or not an item is out of specification for picking work based on at least the size information and weight information of the item.

6. The management system according to claim 1, further comprising a second out-of-specification determination unit that determines that an item is out of specification in a picking operation if the item falls under conditions arbitrarily set regarding out-of-specification.

7. The management system according to claim 1, further comprising a controller which determines whether or not the robot can perform the picking operation on the target item based on the results of the operation verification by the robot.

8. The robot system is equipped with multiple types of robot hands. The management system according to claim 1, wherein the picking feasibility determination unit determines whether or not the robot can perform the picking operation based on the picking feasibility information for each of the multiple types of hands.

9. When at least some of the functions of the aforementioned management system are updated, The management system according to claim 7, wherein at least one of the out-of-specification determination unit, the recognition determination unit, the picking feasibility determination unit, and the actual machine picking feasibility determination unit performs a determination related to the updated function.

10. The robot system is equipped with multiple types of robot hands. The management system according to claim 1, wherein if the non-specification determination unit determines that the picking operation is non-specification for any of the multiple types of the hand, the recognition determination unit and the picking feasibility determination unit do not perform any determinations and determine that the picking operation for the target item is not possible.

11. The management system according to claim 1, wherein if the target article is similar in appearance to an article that has already been determined, the determination by at least one of the out-of-specification determination unit and the recognition determination unit is skipped, and the determination result of at least one of the out-of-specification determination unit and the recognition determination unit of the article that has already been determined is used.

12. The management system according to claim 1, wherein the picking feasibility determination unit is capable of performing a picking feasibility determination in a normal mode, which causes the robot to perform a predetermined operation, as well as a safety mode, which causes the robot to operate at a slower speed than the normal mode.

13. The management system according to claim 1, wherein the functions related to the recognition determination unit are updated, and when the updated recognition determination unit determines whether or not the item can be picked, the determination of the out-of-specification determination unit is skipped.

14. The information processing device in a management system comprising: a robot system including a robot that grasps and picks items; an information processing device that determines whether or not it is possible for the robot to pick an item; and a controller that instructs the robot to pick the item that it has been determined is possible for the robot to pick, A non-specification determination unit determines whether the target item is outside the specifications for the picking operation based on information relating to the target item, A recognition determination unit that determines whether or not the item can be recognized based on an image of the item, The system includes a picking feasibility determination unit that estimates whether the robot can grasp the target item based on data related to the robot's gripping operation, and determines whether the picking operation is feasible. An information processing device that determines whether or not the robot can perform the picking operation on the target item based on the determination results of the non-specification determination unit, the recognition determination unit, and the picking feasibility determination unit.

15. The information processing apparatus according to claim 14, wherein if the determination result in the non-specification determination unit indicates that the item is not non-specification, the determination result in the recognition determination unit indicates that it is recognizable, and the determination result in the picking feasibility determination unit indicates that it is pickable, the apparatus determines that the item can be picked by the robot.

16. The information processing device according to claim 15, wherein the information processing device performs the following determinations in the order of determination by the non-specification determination unit, determination by the recognition determination unit, and determination by the picking feasibility determination unit.

17. A management system comprising: a robot system including a robot that grasps and picks items; an information processing device that determines whether or not it is possible for the robot to pick an item; and a controller that instructs the robot to pick the item that it has been determined is possible for the robot to pick, wherein the information processing device performs information processing in the management system, An out-of-specification determination step in which the out-of-specification determination unit determines whether the target item is out of specification for picking operations based on information belonging to the target item, A recognition determination step in which the recognition determination unit determines whether or not the item can be recognized based on the image of the item, A picking feasibility determination step involves a picking feasibility determination unit estimating, based on data related to the robot's gripping operation, whether or not the robot can grip the target item, and determining whether or not the picking operation is feasible. An information processing method comprising a step of determining, based on the results of the aforementioned non-specification determination step, the recognition determination step, and the picking feasibility determination step, whether or not the robot can perform a picking operation on the target item.

18. The information processing method according to claim 17, wherein if the result of the determination step in the non-specification determination step is that the item is not non-specification, the result of the determination step in the recognition determination step is that it is recognizable, and the result of the determination step in the picking feasibility determination step is that it is pickable, then it is determined that the item can be picked by the robot.

19. The information processing method according to claim 18, wherein each determination is made in the order of determination in the non-specification determination step, determination in the recognition determination step, and determination in the picking feasibility determination step.

20. A computer is the information processing device in a management system comprising: a robot system including a robot that grasps and picks items; an information processing device that determines whether or not it is possible for the robot to pick an item; and a controller that instructs the robot to pick the item that it has been determined is possible for the robot to pick. A non-specification determination unit determines whether the target item is outside the specifications for the picking operation based on information relating to the target item, A recognition determination unit that determines whether or not the item can be recognized based on an image of the item, A picking feasibility determination unit estimates whether the robot can grasp the target item based on data related to the robot's gripping operation and determines whether the picking operation is feasible. A program for functioning as a means for determining whether or not the robot can perform the picking operation on the target item, based on the determination results of the non-specification determination unit, the recognition determination unit, and the picking feasibility determination unit.

21. The program according to claim 20, wherein if the determination result in the non-specification determination unit indicates that the item is not outside the specifications, the determination result in the recognition determination unit indicates that it is recognizable, and the determination result in the picking feasibility determination unit indicates that it is pickable, the program determines that the item can be picked by the robot.

22. The program according to claim 21, wherein the determinations are made in the following order: determination by the non-specification determination unit, determination by the recognition determination unit, and determination by the picking feasibility determination unit.

Images