Picking device and picking method

Abstract

To provide a picking device which can reduce a time for picking work even when failing in picking up an article.SOLUTION: A picking device includes a measurement part 2 which acquires measurement data of a sensor, a recognition part 3 which grasps an article condition by recognizing an article on the basis of the measurement data, an operation part 4 which makes a robot execute pickup operation of the article, an article condition change detection part 5 which detects whether the article condition changes or not when recognition or the pickup operation of the article is failed, and a work environment change part 6 which changes a work environment of picking work. When the article condition change detection part 5 detects the change of the article condition, at least one of acquisition of the measurement data, recognition of the article, and the pickup operation is executed without changing the work environment. When the article condition change detection part 5 does not detect the change of the article condition, at least one of the acquisition of the measurement data, recognition of the article, and pickup operation is executed after the work environment change part 6 changes the work environment.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a picking device for performing an item picking operation and a method for picking an item. [Background technology]

[0002] A picking device is used in industrial facilities such as factories and warehouses, as well as commercial facilities, to operate a robot to pick items. Picking devices are expected to complete the picking operation without interruption, even if they fail to pick up an item. With conventional picking devices, if they fail to pick up an item, they must change the work environment for the picking operation and try the picking operation again.

[0003] An example of a conventional picking device is described in Patent Document 1. The workpiece picking device described in Patent Document 1 includes a sensor that measures a plurality of workpieces (items), a holding position and posture calculation unit that calculates data on the position and posture at which the robot holds the workpieces based on the output of the sensor, a loading condition improvement action generation unit that generates data on actions to improve the loading condition of the workpieces based on the output of the sensor, and a robot control unit that controls the robot and its hand based on the outputs of the holding position and posture calculation unit and the loading condition improvement action generation unit to pick up the workpieces and perform actions to improve the loading condition. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-126802 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, in conventional picking devices such as the device described in Patent Document 1, when an item fails to be picked up, it is necessary to change the work environment of the picking operation (e.g., the state of the item or the picking operation). However, changing the work environment takes time, which increases the processing time of the picking operation and can reduce work efficiency. For this reason, it is desirable to reduce the frequency of changing the work environment of the picking operation even when an item fails to be picked up, thereby reducing the time required for the picking operation.

[0006] An object of the present invention is to provide a picking device and a picking method that can reduce the time required for picking work even if an item fails to be picked up. [Means for solving the problem]

[0007] A picking device according to the present invention includes a robot that performs a picking operation for an item placed in a work area, a measurement unit that is connectable to a measurement sensor that measures the work area and acquires measurement data obtained by the measurement sensor measuring the work area, a recognition unit that recognizes the item based on the measurement data and determines the item state, i.e., the physical state of the item, and an operation unit that causes the robot to perform a picking operation for the item based on the recognition result of the recognition unit, an item state change detection unit that detects whether or not the item state has changed if the recognition unit fails to recognize the item or if the operation unit fails to pick up the item, and a work environment change unit that changes at least one of the item state, the measurement conditions of the measurement sensor, and the picking operation to change the work environment for the picking operation. When the item state change detection unit detects a change in the item state, at least one of the measurement data acquisition by the measurement unit, the recognition of the item by the recognition unit, and the picking operation by the operation unit is performed without changing the work environment. If the item state change detection unit does not detect a change in the item state, the work environment change unit changes the work environment, and then performs at least one of the following: acquisition of the measurement data by the measurement unit, recognition of the item by the recognition unit, and the pickup operation by the operation unit.

[0008] A picking method according to the present invention is a method for using a robot to pick up items placed in a work area, the method comprising: a measurement step of acquiring measurement data obtained by measuring the work area with a measurement sensor; a recognition step of recognizing the item based on the measurement data and determining the item state, i.e., the physical state of the item; an operation step of causing the robot to perform a pick-up operation for the item based on the item recognition result obtained in the recognition step; an item state change detection step of detecting whether or not the item state has changed if the item recognition step fails or the pick-up operation fails in the operation step; and a work environment change step of changing at least one of the item state, the measurement conditions of the measurement sensor, and the pick-up operation to change the work environment for the pick-up operation. If a change in the item state is detected in the item state change detection step, at least one of the measurement step, the recognition step, and the operation step is performed without changing the work environment. If a change in the item state is not detected in the item state change detection step, at least one of the measurement step, the recognition step, and the operation step is performed after changing the work environment. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a picking device and a picking method that can reduce the time required for picking work even if an item fails to be picked up. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a functional configuration of a picking device according to an embodiment of the present invention; [Figure 2] 1 is a block diagram illustrating a hardware configuration of a picking device according to an embodiment of the present invention. [Figure 3] 1 is a schematic diagram showing the configuration of a picking system including a picking device according to an embodiment of the present invention. [Figure 4] 10 is a flowchart of a picking process executed by the picking device according to the present embodiment. [Figure 5]10A and 10B are diagrams illustrating examples of factors that cause a picking operation to fail and changes in the operation environment that correspond to the factors. [Figure 6A] FIG. 10 is a diagram showing an example of items stored in a container before the robot performs a pick-up operation. [Figure 6B] 10A and 10B are diagrams illustrating examples of articles when the robot fails to pick them up. [Figure 6C] 10A and 10B are diagrams illustrating examples of areas where the state of an item has changed, as determined by an item state change detection unit. [Figure 7] 10 is a flowchart of a picking process including a process of checking whether or not the work environment has been changed by a work environment change unit. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the picking device and picking method according to the present invention, a robot is operated to pick up the recognized item. Even if the picking of the item (including the item recognition and the item picking operation) fails, the frequency of changing the work environment for the picking operation can be reduced compared to conventional methods, thereby reducing the time required for the picking operation.

[0012] The work environment for picking operations includes the item state, measurement conditions, and pickup operations. The item state refers to the physical state of the item to be picked up, including, for example, the position, posture, shape, and appearance of the item. The measurement conditions refer to the conditions under which the measurement sensor connected to the picking device measures the item. The pickup operation refers to the operation by which the robot connected to the picking device picks up the item.

[0013] Normally, when the state of an item changes, the recognition result of the item measured by the measurement sensor and the pickup operation of the robot to pick up the item also change automatically in accordance with the change in the state of the item. Therefore, in the present invention, when an item pickup fails, i.e., when the item recognition or pickup operation fails, if the item state has changed, the picking operation is re-executed without changing the work environment for the picking operation, and if the item state has not changed, the work environment is changed and the picking operation is re-executed.

[0014] In other words, in the present invention, the work environment is changed only when an attempt to pick an item fails and the item state has not changed, and the work environment is not changed when an attempt to pick an item fails and the item state has changed. Because the picking device and picking method according to the present invention have such a configuration, it is possible to reduce the frequency of changing the work environment and reduce the time required for picking work even if an attempt to pick an item fails.

[0015] The picking device and the picking method according to the present invention will be described below with reference to the drawings. In the drawings used in this specification, the same or corresponding components are designated by the same reference numerals, and repeated description of these components may be omitted. [Example]

[0016] A picking device and a picking method according to an embodiment of the present invention will be described.

[0017] 1 is a block diagram showing the functional configuration of a picking device according to this embodiment. The picking device 1 according to this embodiment can be configured with a computer, and includes a measurement unit 2, a recognition unit 3, an operation unit 4, an item state change detection unit 5, and a work environment change unit 6, and executes a picking process for items placed in a work area.

[0018] As will be described later, the picking device 1 can be connected to a robot that performs the work of picking items and a measurement sensor that measures the work area where the robot performs the picking work. The picking device 1 operates the robot to perform the picking process of items. The robot receives instructions from the picking device 1 and performs a pickup operation to pick up the items.

[0019] The measurement unit 2 acquires data obtained by measuring the work area using a measurement sensor connected to the picking device 1, i.e., measurement data of the work area. The measurement sensor is a sensor that measures objects (including items to be picked up by the robot) present in the work area, and can be configured with at least one of an image sensor such as a camera, a distance sensor that measures distance, and a tactile sensor, for example.

[0020] For example, if the measurement sensor is an image sensor, the measurement unit 2 acquires images such as RGB images, gray images, and infrared images as measurement data. Also, for example, if the measurement sensor is a distance sensor (e.g., LiDAR, laser displacement meter, stereo camera, infrared rangefinder, laser rangefinder, and TOF range sensor), the measurement unit 2 acquires point cloud data measured by the measurement sensor as measurement data. Also, for example, if the measurement sensor is a tactile sensor, the measurement unit 2 acquires tactile data measured by the measurement sensor as measurement data.

[0021] The recognition unit 3 recognizes an item placed in the work area based on the measurement data of the work area 60 acquired by the measurement unit 2. By recognizing the item, the recognition unit 3 can determine what the item is and its condition. The recognition unit 3 can recognize the item using existing technology. Examples of methods for item recognition by the recognition unit 3 include feature point matching using methods such as SIFT and ORB, deep learning using methods such as R-CNN, YOLO, and SSD, segmentation using methods such as Mask R-CNN and GAN, and grip point recognition using methods such as Dex-Net.

[0022] The operation unit 4 causes the robot to perform a pickup operation to pick up an item placed in the work area based on the item recognition result by the recognition unit 3. The robot has a hand, and when connected to the picking device 1 and performs a pickup operation, the robot picks up an item with the hand.

[0023] The item state change detection unit 5 detects whether or not there has been a change in the item state when the recognition unit 3 fails to recognize the item, or when the operation unit 4 fails to perform the robot pick-up operation.

[0024] The work environment change unit 6 changes at least one of the item state, the measurement conditions of the measurement sensor, and the picking operation by the operation unit 4 to change the work environment for the picking operation.

[0025] 2 is a block diagram showing the hardware configuration of the picking device 1. The picking device 1 includes a memory 101, a storage device 102, a processor 100, an input device 103, and an output device 104. The memory 101, the storage device 102, the processor 100, the input device 103, and the output device 104 are connected to one another by a transmission path (bus) 106.

[0026] The memory 101 stores a program P1 for the picking process executed by the picking device 1.

[0027] The storage device 102 includes a data storage unit R1, and stores data calculated by the picking device 1 and data acquired by the picking device 1 from the robot and measurement sensors in the data storage unit R1.

[0028] The processor 100 performs calculations using the picking process program P1 and data stored in the data storage unit R1, and causes the picking device 1 to execute the picking process.

[0029] The input device 103 is an interface for inputting data from the robot and the measurement sensor. The input device 103 can also input data from other devices connected to the picking device 1.

[0030] The output device 104 is an interface that outputs instructions and data to the robot and the measurement sensor. The output device 104 can also output data to other devices connected to the picking device 1.

[0031] The picking device 1 can be equipped with a display unit 105. The display unit 105 is connected to a transmission path 106 and displays information such as data input by the picking device 1 and calculation results of the picking device 1. The display unit 105 may be a device that can be connected to the picking device 1 but is different from the picking device 1.

[0032] FIG. 3 is a schematic diagram showing the configuration of a picking system 50 including a picking device 1 according to this embodiment. The picking system 50 includes the picking device 1 according to this embodiment, a robot 20, and a measurement sensor 30. An item 40 stored in a container 41 is present in a work area 60 where the robot 20 performs picking work. In the example shown in FIG. 3, the container 41 is placed on a platform 42. The robot 20 picks up the item 40. Note that the item 40 does not have to be stored in the container 41.

[0033] The picking device 1 can be connected to a measurement sensor 30 and a robot 20, acquires measurement data from the measurement sensor 30, and operates the robot 20 to perform a picking process for an item 40.

[0034] The measurement sensor 30 measures objects present in the work area 60 and acquires measurement data of the work area 60. The work area 60 includes items 40, containers 41, and other objects. The measurement sensor 30 can be configured with at least one of an image sensor including a camera, a distance sensor, and a tactile sensor, for example. The measurement sensor 30 can also be equipped with a lighting device that illuminates the work area 60.

[0035] The robot 20 has a hand 21, and performs a picking operation of picking up an item 40 with the hand 21. The robot 20 can be configured as any robot, such as an arm robot, a parallel link robot, or an orthogonal robot. The hand 21 has at least one of a finger portion having multiple fingers and a suction portion, and picks up the item 40 by gripping the item 40 with the finger portion or by suctioning the item 40 with the suction portion.

[0036] The picking device 1 can change the orientation and position of the measurement sensor 30 and change the distance between the measurement sensor 30 and the item 40. The picking device 1 can also shake the container 41 that stores the item 40 (i.e., impart vibrations to the container 41). The picking device 1 can operate devices (e.g., actuators, vibrators, etc.) that can move the measurement sensor 30, the item 40, and the container 41.

[0037] 4 is a flowchart of the picking process executed by the picking device 1 according to this embodiment. The picking device 1 executes the picking method according to this embodiment in accordance with the flowchart shown in FIG.

[0038] In S1 , the measurement unit 2 acquires measurement data of the working area 60 from the measurement sensor 30 .

[0039] In S2, the recognition unit 3 recognizes the item 40 placed in the work area 60 based on the measurement data of the work area 60 acquired by the measurement unit 2. By recognizing the item 40, the recognition unit 3 can determine what the item 40 is and the item state. The item state is the physical state of the item 40, and includes, for example, the position, posture, shape, and appearance of the item 40.

[0040] In S3, the recognition unit 3 determines whether the recognition of the article 40 was successful or unsuccessful. The recognition unit 3 can determine that the recognition of the article 40 was unsuccessful, for example, when it did not recognize any article 40, when it did not output the position or orientation of the recognized article 40, when it recognized an article 40 whose size was equal to or smaller than a predetermined threshold, when it recognized multiple articles 40 in the same area, or when the reliability (reliability score) of the recognition result of the article 40 was smaller than a predetermined threshold.

[0041] If the recognition unit 3 succeeds in recognizing the article 40, the process of S4 is executed. If the recognition unit 3 fails to recognize the article 40, the process of S6 is executed.

[0042] In S4, the operation unit 4 causes the robot 20 to perform a pickup operation to pick up the item 40 placed in the working area 60, based on the item recognition result by the recognition unit 3. The operation unit 4 knows the item state (e.g., the position, posture, shape, and appearance of the item 40) from the item recognition result by the recognition unit 3, so it calculates the trajectory along which the hand 21 will move in accordance with the item state, and causes the robot 20 to perform the pickup operation. The robot 20 moves the hand 21 along the calculated trajectory, and performs the pickup operation to pick up the item 40 with the hand 21.

[0043] In S5, the operation unit 4 determines whether the pickup operation of the robot 20 was successful or unsuccessful. The operation unit 4 can determine that the pickup operation of the robot 20 failed, for example, when it cannot calculate the trajectory of the hand 21 that can reach the item 40, when the tactile sensor detects that the item 40 has collided with the robot 20 or another object present in the working area 60, when the tactile sensor detects that the picked-up item 40 has slipped, tilted, or fallen, when the weight sensor detects that the weight of the picked-up item 40 is above or below a predetermined threshold, or when the pressure of the suction part of the hand 21 during suction and movement changes beyond a threshold. The operation unit 4 can also determine whether the pickup operation of the robot 20 was successful or unsuccessful using an image captured of the pickup operation of the robot 20.

[0044] If the pick-up operation of the robot 20 by the operation unit 4 fails, the process of S6 is executed.

[0045] In S6, the item state change detection unit 5 detects whether the item state has changed. The process of S6 is executed when the picking device 1 fails to pick up the item 40. That is, the process of S6 is executed when the recognition unit 3 fails to recognize the item, and when the operation unit 4 fails to perform the pick-up operation of the robot 20.

[0046] In S6, the measurement unit 2 acquires new measurement data of the work area 60 from the measurement sensor 30, and the recognition unit 3 re-recognizes the item 40 placed in the work area 60 based on the measurement data of the work area 60 newly acquired by the measurement unit 2, and acquires the item state (second item state). The item state change detection unit 5 acquires from the recognition unit 3 the item state (second item state) acquired when the recognition unit 3 re-recognizes the item 40, and compares this second item state (item state acquired in S6) with the item state acquired in S2 (first item state). If the result of this comparison shows that the item state acquired in S6 differs from the item state acquired in S2, the item state change detection unit 5 detects that the item state has changed.

[0047] The item state change detection unit 5 can also compare the measurement data (second measurement data) of the work area 60 newly acquired by the measurement unit 2 with the measurement data (first measurement data) of the work area 60 acquired by the measurement unit 2 in S1, and if the result of this comparison shows that the second measurement data is different from the first measurement data, detect that the item state has changed. The item state change detection unit 5 can also compare the second measurement data with the item state (first item state) acquired by the recognition unit 3 in S2, and if the result of this comparison shows that the second measurement data is different from the first item state, detect that the item state has changed.

[0048] In S6, the measurement data newly acquired by the measurement unit 2 can include, in addition to the data already mentioned, data on the pressure when the hand 21 of the robot 20 picks up the item 40, and the amount of movement of the belt conveyor when the belt conveyor transporting the item 40 is driven (the amount of movement of the item 40 transported by the belt conveyor).

[0049] In addition, if the recognition unit 3 fails to recognize the item 40 in at least one of S2 and S6 and is unable to grasp the item state and compare the item states, the item state change detection unit 5 determines that the item state has changed.

[0050] The item state change detection unit 5 compares the item state obtained in S6 with the item state obtained in S2 and detects a change in the item state in the following cases, for example: If a predetermined number or more pixels whose brightness or RGB values ​​have changed exist in the image captured by the image sensor, it is determined that the appearance of the item 40 or the ambient light has changed, and a change in the item state is detected. Furthermore, if a predetermined number or more points whose distances have changed exist in the point cloud measured by the distance sensor, it is determined that an object other than the item 40 in the working area 60 has been removed or the measurement noise has changed, and a change in the item state is detected. Furthermore, by lifting the item 40 with a tactile sensor, measuring the pressure when the robot 20's hand 21 picks up the item 40, or determining the center of gravity of the container 41 containing the item 40, it is possible to determine changes in the position, posture, or shape of the item 40 due to, for example, a drop or collapse. If any of these changes exist, a change in the item state is detected. Furthermore, if deformation of the container 41 containing the item 40 is detected, a change in the item state is detected. Furthermore, when the belt conveyor that transports the article 40 is driven, it can also be assumed that a change in the state of the article is detected.

[0051] As shown in S7, if the item state change detection unit 5 detects a change in the item state in S6, the process returns to S1, and without changing the work environment for the picking work, the measurement unit 2 acquires measurement data, the recognition unit 3 recognizes the item, and the operation unit 4 performs the pick-up operation of the robot 20 again. If the item state change detection unit 5 does not detect a change in the item state in S6, the process executes the processing of S8 to change the work environment for the picking work, and then the process returns to S1, and the measurement unit 2 acquires measurement data, the recognition unit 3 recognizes the item, and the operation unit 4 performs the pick-up operation of the robot 20 again.

[0052] Note that if the measurement unit 2 acquires new measurement data of the work area 60 in S6, the process may return from S7 to S1 and execute the process of S2 without executing the process of S1. In other words, if the item state change detection unit 5 detects a change in the item state in S6, the process may return from S7 to S2 and re-execute recognition of the item 40. Figure 4 shows an example of a flowchart for returning from S7 to S1.

[0053] For example, if in S6 the item state change detection unit 5 detects a change in the item state from physical information about the item 40, it is possible to return from S7 to S1 and execute the process of S1. An example of this is described below. In S6, if the item state change detection unit 5 detects a change in the item state by lifting the item 40 with a tactile sensor or measuring the pressure when the hand 21 of the robot 20 picks up the item 40 and determines that the position, posture, or shape of the item 40 has changed, it is possible to return from S7 to S1. Also, if in S6 a change in the item state is detected when the center of gravity of the container 41 containing the item 40 has changed or when the belt conveyor transporting the item 40 has started to move, it is possible to return from S7 to S1.

[0054] Also, for example, if there is a change in the measurement data acquired by the measurement sensor 30 in S6, this measurement data can be used to execute the process of S2 (recognizing the article 40), so it is possible to return from S7 to S2. For example, if there is a predetermined number or more pixels in the image captured by the image sensor whose brightness or RGB value has changed in S6, or if there is a predetermined number or more points in the point cloud measured by the distance sensor whose distance has changed, and thus the article state change detection unit 5 detects a change in the article state, it is possible to return from S7 to S2.

[0055] In S8, the work environment change unit 6 changes the work environment for the picking operation. As already mentioned, the work environment for the picking operation includes the item state, measurement conditions, and pick-up operation. The process of S8 is executed when the picking device 1 fails to pick the item 40, i.e., when the recognition unit 3 fails to recognize the item or the operation unit 4 fails to pick up the robot 20, and the item state change detection unit 5 does not detect a change in the item state.

[0056] The work environment changing unit 6 can change any work environment as long as it corresponds to the cause of the failure of the picking operation of the item 40. For example, the work environment changing unit 6 changes at least one of the item state, the measurement conditions of the measurement sensor 30, and the picking operation of the robot 20 by the operation unit 4 to change the work environment for the picking operation. The work environment changing unit 6 can change the work environment by selecting the work environment in order of shortest processing time from the work environments corresponding to the cause of the failure of the picking operation, or by randomly selecting the work environment.

[0057] When the process of S8 is completed, the process returns to S1 to continue the picking process. Note that, depending on the work environment changed by the work environment change unit 6, even if the process returns from S8 to S1, the process of S4 may be executed without executing the processes of S1 to S3. In other words, when the process of S8 is completed, the process may return from S8 to S4 to continue the picking process. FIG. 4 shows an example of a flowchart for returning from S8 to S1.

[0058] Next, we will explain examples of factors that cause picking work to fail and changes to the work environment that correspond to these factors.

[0059] 5 is a diagram showing examples of factors that cause a picking operation to fail and changes to the work environment that correspond to these factors. As described above, picking operation failures include failure to recognize an item and failure to pick up the item.

[0060] Examples of failures in item recognition include missing measurement points in the point cloud data measured by the measurement sensor 30 (distance sensor), a small, few, or misaligned area of ​​the measured point cloud, and large point cloud noise. Examples of failures in item recognition include images captured by the measurement sensor 30 (image sensor) that are dark or bright, and the contours of objects captured in the images that are unclear. Examples of failures in item recognition include when the shape of the item 40 recognized by the recognition unit 3 is small in surface size, when the packaging (e.g., a bag) of the item 40 is wrinkled, or when the surface shape is more complex than expected.

[0061] In response to such a failure to recognize an object, the work environment change unit 6 can change the work environment as follows. That is, the work environment can be changed by changing the position and orientation of the measurement sensor 30, changing the type of measurement sensor 30, changing the position and orientation of the lighting device provided in the measurement sensor 30, and changing the position and orientation of the object 40. More specifically, the work environment can be changed, for example, by changing the orientation or position of the measurement sensor 30 or by changing the distance between the measurement sensor 30 and the object 40. The work environment can also be changed by, for example, shaking the container 41 storing the object 40, or by shifting, lifting, pulling, or tipping the object 40, thereby changing the position and orientation of the object 40. The work environment can also be changed, for example, by blowing air to widen the gaps between multiple objects 40, thereby changing the position and orientation of the objects 40.

[0062] Examples of failed pickup operations include when the trajectory of the hand 21 of the robot 20 collides with an object other than the item 40, when it is blocked by an object other than the item 40 and cannot get close to the item 40, when the moving distance is too short to reach the item 40, and when it cannot reach the gripping point of the item 40 and cannot pick up the item 40. Examples of failed pickup operations include when the hand 21 is unable to grip or adsorb the item 40, when the picked-up item 40 slips, tilts, or falls, and when the weight of the picked-up item 40 is above or below a predetermined threshold and therefore cannot pick up the item 40.

[0063] In response to such a failed pickup operation, the work environment change unit 6 can change the work environment as follows: That is, the work environment can be changed by changing the position and posture of the item 40, changing the position and posture of the robot 20, changing the hand 21 of the robot 20, and changing the pickup position of the item 40. More specifically, the work environment can be changed by, for example, changing the position of the robot 20 relative to the item 40, changing the fingers of the hand 21, or changing the position where the hand 21 grasps the item 40.

[0064] In this way, the work environment change unit 6 changes at least one of the item state (e.g., the position and posture of the item 40), the measurement conditions of the measurement sensor 30 (e.g., the position and posture of the measurement sensor 30), and the pick-up operation of the robot 20 by the operation unit 4 (e.g., the position and posture of the robot 20), thereby changing the work environment for the picking work.

[0065] As described above, once the processing of S8 in Fig. 4 is completed, the process may return from S8 to S1 or S4, depending on the work environment changed by the work environment change unit 6. For example, if the change in the work environment is related to the measurement sensor 30 or a change in the position or posture of the item 40, the process may return to S1 once the processing of S8 is completed. Also, for example, if the change in the work environment is a change in the pickup position of the item 40 or a change in the hand 21, the process may return to S4 once the processing of S8 is completed. Also, for example, if the change in the work environment is a change in the position of the robot 20 relative to the item 40, the process may return to S1 or S4 once the processing of S8 is completed.

[0066] When the item state change detection unit 5 detects a change in the item state, it can determine the area (partial area) where the item state has changed within the working area 60. The item state change detection unit 5 determines any area within the working area 60 that includes the item 40 whose item state has changed, and sets the determined area as the partial area where the item state has changed.

[0067] 6A is a diagram showing an example of an item 40 stored in a container 41 before the robot 20 performs a pick-up operation. FIG. 6A shows the item 40 as seen from above. In the working area 60, there are a plurality of items 40 stored in the containers 41. The robot 20 is to pick up item 40a among the items 40.

[0068] 6B is a diagram showing an example of the item 40 when the robot 20 fails to pick up the item 40. Assume that the robot 20 drops the picked-up item 40a from the hand 21 during the pickup operation, causing the pickup operation to fail. The item 40a has fallen from the hand 21 and is now located on top of the item 40, and the item state (position and posture in the example shown in FIG. 6B) has changed from the state shown in FIG. 6A.

[0069] FIG. 6C is a diagram showing an example of a partial area 45, which is an area where the item state has changed, determined by the item state change detection unit 5. The item state change detection unit 5 determines an area including the item 40a where the item state has changed as the partial area 45 where the item state has changed. The partial area 45 is a part of the working area 60. The item state change detection unit 5 can arbitrarily determine the size and shape of the partial area 45.

[0070] After the item state change detection unit 5 detects a change in the item state, the recognition unit 3 performs processing to recognize the item 40 for the partial area 45 determined by the item state change detection unit 5 in S2 of the flowchart shown in Fig. 4. The recognition unit 3 re-executes processing to recognize the item 40 only for the partial area 45 where the item state has changed, thereby reducing the time required for the recognition processing of the item 40 and the time required for the picking operation.

[0071] In this embodiment, after the work environment change unit 6 changes the work environment for the picking work, it is possible to check whether the work environment has changed. When the work environment is changed, the item status may change. The item status change detection unit 5 detects whether the item status has changed due to the change in the work environment, thereby making it possible to check whether the item status has changed and the work environment has changed.

[0072] 7 is a flowchart of the picking process executed by the picking device 1 according to this embodiment, and includes a process for checking whether the work environment has been changed by the work environment change unit 6. In the flowchart shown in FIG. 4, once the process of S8 is completed, the process returns from S8 to S1 (or S4). In the flowchart shown in FIG. 4, once the process of S8 is completed, the process returns from S8 to S6.

[0073] In S8, the work environment change unit 6 changes the work environment for the picking work. After the process of S8 is completed, the process returns to S6, and the item state change detection unit 5 detects whether the item state has changed.

[0074] For example, if multiple items 40 are tightly packed in a container 41, even if the work environment change unit 6 attempts to change the work environment by shaking the container 41 in S8, the item states will not change and the work environment cannot be changed. In this case, in S6, the item state change detection unit 5 detects that the item states have not changed. Then, in S8, the work environment change unit 6 can change the work environment by another method (for example, by changing the distance between the measurement sensor 30 and the item 40).

[0075] By performing the process of the flowchart shown in FIG. 7, it is possible to effectively check whether the work environment has been changed after the work environment change unit 6 has changed the work environment for the picking work.

[0076] Furthermore, in this embodiment, if the item status does not change even when the work environment change unit 6 changes the work environment for the picking work, the user of the picking device 1 can be notified that the item status has not changed.

[0077] The display unit 105 of the picking device 1 displays work environment change failure information indicating that the work environment change unit 6 has failed to change the work environment in at least one of the following cases after the work environment change unit 6 has changed the work environment for the picking work in S8: when the recognition unit 3 has failed to recognize the item 40 in S2; when the robot 20 has failed to pick the item 40 in S4; and when the item state change detection unit 5 has detected that the item state has not changed in S6. The picking device 1 may output work environment change failure information to the display unit 105 when the change of the work environment has failed a predetermined number of times or more.

[0078] The information about the failure of the work environment change can include the history of the work environment change performed by the work environment change unit 6 and the detection result of the change in the item status performed by the item status change detection unit 5. That is, the display unit 105 can display at least one of the history of the work environment change and the detection result of the change in the item status (for example, the change in the item status detected by the item status change detection unit 5, or the fact that the item status change detection unit 5 did not detect a change in the item status) in addition to the failure to change the work environment.

[0079] By having the picking device 1 output the history of work environment changes as work environment change failure information to the display unit 105, the user can recognize a work environment change that does not change the item status and can cancel such a work environment change. Also, by having the picking device 1 output the detection result of the item status change as work environment change failure information to the display unit 105, the user can obtain information that is useful for identifying the cause of the failure in repeatedly recognizing and picking the item 40 despite detecting a change in the item status. Therefore, the picking device 1 does not repeatedly make unnecessary changes to the work environment that do not change the item status, and can reduce the time required for picking work.

[0080] As described above, the picking device and picking method according to this embodiment change the work environment for the picking operation and perform the pickup operation only if the pickup of the item 40 fails (i.e., if the item recognition or pickup operation fails) and the item status has not changed. If the pickup of the item 40 fails and the item status has changed, the pickup operation is performed without changing the work environment. Therefore, the picking device and picking method according to this embodiment can reduce the frequency of changing the work environment and reduce the time required for the picking operation even if the pickup of the item 40 fails.

[0081] The present invention is not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to embodiments including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete part of the configuration of each embodiment, or to add or replace other configurations. [Explanation of symbols]

[0082] 1...Picking device, 2...Measurement unit, 3...Recognition unit, 4...Operation unit, 5...Item state change detection unit, 6...Work environment change unit, 20...Robot, 21...Hand, 30...Measurement sensor, 40, 40a...Item, 41...Container, 42...Table, 45...Partial area, 50...Picking system, 60...Work area, 100...Processor, 101...Memory, 102...Storage device, 103...Input device, 104...Output device, 105...Display unit, 106...Transmission path.

Claims

1. A robot that performs a picking operation on an item placed in a work area can be connected to a measurement sensor that measures the work area, a measurement unit that acquires measurement data obtained by measuring the working area with the measurement sensor; a recognition unit that recognizes the item based on the measurement data and grasps the item state, which is the physical state of the item; an operation unit that causes the robot to perform an operation of picking up the item based on the result of the recognition of the item by the recognition unit; an item state change detection unit that detects whether or not there is a change in the item state when the recognition unit fails to recognize the item or when the operation unit fails to perform the pickup operation; a work environment change unit that changes at least one of the item state, the measurement conditions of the measurement sensor, and the pick-up operation to change the work environment of the picking operation; Equipped with When the item state change detection unit detects a change in the item state, at least one of the following is executed without changing the working environment: acquisition of the measurement data by the measurement unit; recognition of the item by the recognition unit; and pickup operation by the operation unit; When the item state change detection unit does not detect a change in the item state, the work environment change unit changes the work environment, and then performs at least one of the following: acquisition of the measurement data by the measurement unit; recognition of the item by the recognition unit; and the pick-up operation by the operation unit. A picking device characterized by:

2. The item state includes the position, posture, shape, and appearance of the item. The picking device according to claim 1 .

3. the measurement unit newly acquires the measurement data when the item state change detection unit detects whether or not there is a change in the item state; when the item state change detection unit detects whether or not there is a change in the item state, the recognition unit recognizes the item again based on the measurement data newly acquired by the measurement unit to grasp a second item state; The item state change detection unit detects whether or not the item state has changed by comparing the second item state with the first item state, which is the item state recognized by the recognition unit before the pickup operation by the operation unit. The picking device according to claim 1 .

4. the measurement unit newly acquires the measurement data as second measurement data when the item state change detection unit detects whether or not the item state has changed; the item state change detection unit detects whether or not the item state has changed by comparing the second measurement data with the first measurement data, which is the measurement data acquired by the measurement unit before the pick-up operation by the operation unit. The picking device according to claim 1 .

5. the work environment change unit changes the work environment by changing at least one of the position of the measurement sensor, the attitude of the measurement sensor, the position of the item, the attitude of the item, the position of the robot, the attitude of the robot, the type of the measurement sensor, and a hand provided on the robot for picking up the item. The picking device according to claim 1 .

6. When the item state change detection unit detects a change in the item state, it determines a partial area in the work area where the item state has changed, and the recognition unit recognizes the item in the partial area after the item state change detection unit detects the change in the item state. The picking device according to claim 1 .

7. the item state change detection unit detects whether or not the item state has changed after the work environment change unit changes the work environment; The picking device according to claim 1 .

8. A display unit is provided, the display unit displays failure information that the work environment change unit has failed to change the work environment in at least one of the following cases: after the work environment change unit has changed the work environment, the recognition unit has failed to recognize the item, the robot has failed to perform the picking work, and the item state change detection unit has detected that the item state has not changed. The picking device according to claim 7.

9. the display unit displays a history of changes to the work environment made by the work environment change unit. The picking device according to claim 8.

10. the display unit displays the detection result of the change in the item state by the item state change detection unit. The picking device according to claim 8.

11. A method for performing a picking operation on an item placed in a work area by a robot, comprising: a measurement step of acquiring measurement data obtained by measuring the working area with a measurement sensor; a recognition step of recognizing the item based on the measurement data and grasping the item state, which is the physical state of the item; an operation step of causing the robot to perform an operation of picking up the item based on a result of the recognition of the item in the recognition step; an item state change detection step of detecting whether or not there is a change in the item state when the recognition step fails to recognize the item or the pickup operation fails in the operation step; a work environment changing step of changing at least one of the item state, the measurement conditions of the measurement sensor, and the picking operation to change the work environment of the picking operation; and When a change in the state of the item is detected in the item state change detection step, at least one of the measurement step, the recognition step, and the operation step is executed without changing the working environment; If the change in the item state is not detected in the item state change detection step, the work environment is changed in the work environment change step, and then at least one of the measurement step, the recognition step, and the operation step is executed. A picking method characterized by:

Images