Image processing device

Abstract

Provided is an image processing device 1 capable of suppressing the number of times an image of an object 101 needs to be captured while minimizing the risk associated with removing the object 101. The image processing device 1 comprises: an image acquisition unit 11 that acquires image information including height information; a detection unit 13 that performs a detection process for detecting objects 101 from the image information on the basis of information representing characteristics of the objects 101; and a tier setting unit 14 that sets, as a top tier for taking out, objects 101 the top surfaces 102 of which are located within a threshold range below the top surface 102 of the object 101 located at the highest position among the objects 101 detected by the detection unit 13.

Description

Image Processing Device

[0001] The present disclosure relates to an image processing device.

[0002] There is a known depalletizing system in which stacked objects are detected by a camera and then removed by a robot. For example, Japanese Patent Application Laid-Open No. 2003-232662 (Patent Document 1) discloses a technology for acquiring three-dimensional information of an object with high accuracy and high speed based on the distance between the object and an imaging device. Japanese Patent Application Laid-Open No. 2003-232662 (Patent Document 2) discloses a depalletizing device with an accurate work height determination function.

[0003] JP 2012-2683 A JP 4-244391 A

[0004] The problem to be solved by the present disclosure is to provide an image processing device that can reduce the number of times an object is imaged while minimizing the risk of removing the object.

[0005] The image processing device of the present disclosure includes an image acquisition unit that acquires image information including height information, a detection unit that performs a detection process to detect the object from the image information based on information representing the characteristics of the object, and a tier setting unit that sets the object whose top surface is located within a threshold range below the top surface of the object that is highest among the objects detected by the detection unit as the top tier for removal.

[0006] According to the image processing device of the present disclosure, it is possible to provide an image processing device that can reduce the number of times an object is imaged while minimizing the risk of removing the object.

[0007] FIG. 1 is a diagram showing a depalletizing system according to a first embodiment of the present disclosure; FIG. 2 is a diagram for explaining tier setting, as viewed from the side of the stack; FIG. 3 is a flowchart showing the flow of object removal according to the first embodiment; FIG. 4 is a diagram for explaining tier setting according to a second embodiment of the present disclosure, as viewed from the side of the stack; FIG. 5 is a diagram showing a depalletizing system according to the second embodiment; FIG. 6 is a diagram for explaining the object removal sequence according to the second embodiment, as viewed from the side of the stack; FIG. 7 is a flowchart showing the flow of object removal according to the second embodiment; FIG. 8 is a diagram showing a depalletizing system according to a third embodiment of the present disclosure; FIG. 9 is a flowchart showing the flow of object removal according to the third embodiment;

[0008]

[0023] The present disclosure will be described with reference to the accompanying drawings. The present disclosure is not limited to the following embodiments. The present disclosure can be appropriately modified and implemented without departing from the spirit and scope of the present disclosure.

[0009] (Embodiment 1) A depalletizing system 200 according to embodiment 1 of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an overview of the depalletizing system 200 according to embodiment 1. (Outline of Depalletizing System) As shown in Fig. 1, the depalletizing system 200 includes a control unit 10, a visual sensor 30, and a robot 40. The depalletizing system 200 is a system that transports an object 101 to a predetermined location. The object 101 is transported by the transport robot 40.

[0010] An example of the object 101 is a box. A collection of the objects 101 formed by stacking a plurality of the objects 101 is called a stack 100. The objects 101 are placed in the form of the stack 100.

[0011] The control unit 10 controls the visual sensor 30 and the robot 40. The robot 40 is controlled based on image information acquired by the visual sensor 30. These will be described in order below.

[0012] (Visual Sensor) The visual sensor 30 is a device that captures an image within its field of view to acquire image information. An arrow 300 in Fig. 1 indicates an upward direction 300. The visual sensor 30 is disposed above the stack 100 300. The visual field of the visual sensor 30 includes the stack 100.

[0013] The visual sensor 30 can acquire information in the height direction. The image information includes height information of the object 101 included in the stack 100.

[0014] The visual sensor 30 can obtain information about the height direction of the upper surface 102 of at least the object 101 whose upper surface 102 can be recognized when viewing the stacked structure 100 from above. Examples of the visual sensor 30 include a camera, an infrared sensor, and a laser sensor.

[0015] (Robot) The robot 40 is a device that transports the objects 101 to a predetermined location. The stack 100 is arranged within the robot's operating range. The robot 40 includes an arm 41. The robot 40 uses the arm 41 to pick up the objects 101 from the stack 100 and move the objects 101 to a predetermined location. The robot 40 moves the objects 101 one by one.

[0016] (Control Unit) The control unit 10 is a device that controls the visual sensor 30 and the robot 40. The control unit 10 acquires image information from the visual sensor 30. The control unit 10 controls the robot 40 based on the acquired image information.

[0017] The control unit 10 includes an image processing device 1. The image processing device 1 is a device that acquires image information from a visual sensor 30 and sets a step that serves as a basis for the control unit 10 to control the robot 40. The control unit 10 includes an image acquisition unit 11, a height acquisition unit 12, a detection unit 13, a step setting unit 14, a step number output unit 15, and a step threshold setting unit 16.

[0018] The image acquisition unit 11 is a part that acquires image information from the visual sensor 30 or the like. The height acquisition unit 12 is a part that acquires the height of the object 101. The height acquisition unit 12 acquires, in particular, the height of the upper surface 102 of the object 101. The detection unit 13 is a part that performs detection processing. The detection processing is processing that detects the object 101 from the image information based on information that represents the characteristics of the object 101. Examples of the information that represents the characteristics of the object 101 include the shape and size of the object 101.

[0019] The step setting unit 14 is a part that sets a step as the topmost step, for all objects 101 whose upper surfaces 102 are located within a threshold range below the upper surface 102 of the object 101 whose upper surface 102 is at the highest position among the objects 101 detected by the detection unit 13. The step number output unit 15 is a part that outputs the number of steps set by the step setting unit 14 in a format that can be used by the program of the robot 40. The step threshold setting unit 16 is a part that sets the step threshold. The step threshold is the height of one step when the step setting unit 14 sets the step.

[0020] (Layers) In the depalletizing system 200 of this embodiment, tiers are set for the objects 101. The tiers here are tiers that define how the objects 101 are removed. The objects 101 are removed in tier units. Setting of tiers will be described with reference to FIG. 2. FIG. 2 is a diagram illustrating how tiers are set for the objects. The stack shown in FIG. 2 is referred to as stack 110. The objects 101 included in stack 110 are referred to as objects 111 to 115. The direction perpendicular to the upper side 300 is referred to as the lateral direction. FIG. 2 is a view of the stack 110 and the objects 111 to 115 included in stack 110, viewed from the lateral direction. FIG. 2 does not show the front and rear positions relative to the paper surface. FIG. 2 does not show, for example, that objects 111 and 112 are located at the same depth relative to the paper surface. The order from object 111 to object 115 is from the object whose upper surface 102 is located at the top to the object whose upper surface 102 is located at the bottom.

[0021] In the image processing device 1 of this embodiment, among the objects 101 detected by the detection unit 13, all objects 101 whose upper surfaces 102 are located within a range of a threshold value 310 below the upper surface 102 of the object 101 whose upper surface 102 is at the highest position are set as the topmost tier for extraction. This tier setting is performed by the tier setting unit 14. Here, the threshold value 310 refers to the height of one tier, i.e., the length 300 above one tier.

[0022] Arrows 320 in Figure 2 indicate the stage numbers. Three stages, stage 1 to stage 3, are shown in Figure 2.

[0023] An example of setting the tiers will be described with reference to the specific example shown in Fig. 2. (First tier) As described above, all objects within a range of a predetermined threshold value 310 of one tier height from the top surface 102 of the object whose top surface 102 is at the highest position among the detected objects are set to the top tier. In the example shown in Fig. 2, the object at the highest position among the detected objects 111 to 115 is object 111. Therefore, all objects within a range of a predetermined threshold value 310 of one tier height from the top surface 102 of object 111 are set to the top tier. In the example shown in Fig. 2, objects 111 and 112 are set to the first tier.

[0024] (Second tier) The number of tiers is also set for objects below the top tier as follows. Of the objects below the top tier (first tier), an object that is within the range of the threshold 310 for the height of one tier from the top surface 102 of the object whose top surface 102 is at the highest position is set to the second tier. In the example shown in FIG. 2, of the objects below the first tier, the object whose top surface 102 is at the highest position is object 113. Therefore, all objects that are within the range of the threshold 310 for the height of one tier from the top surface 102 of object 113 are set to the second tier. In the example shown in FIG. 2, objects 113 and 114 are set to the second tier.

[0025] The third and subsequent rows are set in the same manner. This process is repeated for all detected objects to set rows. When there are no more objects for which rows have not been set, the row setting is completed.

[0026] The flow of processing will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of object removal in embodiment 1. In the drawings and the following description, S10 indicates step 10. The same applies to the other steps. The flow starts from S10 and ends at S20.

[0027] (S10) S10 is the starting step of the flow. The flow starts at S10.

[0028] (S11) S11 is a step of acquiring a two-dimensional image and a distance image. In S11, the image acquisition unit 11 of the image processing device 1 acquires a two-dimensional image and a distance image of the object 101 included in the stack 100 from the visual sensor 30. The two-dimensional image and the distance image are examples of image information.

[0029] Furthermore, the height acquisition unit 12 acquires the height of the object 101 from the two-dimensional image and the range image. The height acquisition unit 12 acquires, in particular, the height of the upper surface 102 of the object 101.

[0030] (S12) S12 is a detection step. In S12, the detection unit 13 performs a detection process to detect the object 101 from the image information based on information representing the characteristics of the object 101. Note that the characteristics of the object 101 required to detect the object 101 are taught to the detection unit 13 in advance.

[0031] (S13) S13 is a step for determining whether the number of detections is 1 or more. In S13, it is determined whether the number of objects 101 detected in S12 is 1 or more. A detection number of 1 or more indicates that there is an object 101 that has not been taken out. If the detection number is 1 or more, the process proceeds to step S15.

[0032] On the other hand, if the number of detections is not 1 or more, it indicates that there is no unremoved object 101. If the number of detections is not 1 or more, the process proceeds to step S20, where the flow ends.

[0033] (S14) S14 is the step where the setting of the first stage starts (n=1). First, the setting is made from the first stage. The specific processing contents will be described from S15 onwards.

[0034] (S15) S15 is a step in which a box whose top surface is located within the range of the [tier threshold] below the top surface of the highest box (object) for which a tier number has not been set is designated as the nth tier. Here, n = 1. In S15, the tier setting unit 14 designates a box whose top surface is located below the top surface of the object whose top surface is located at the highest position as the first tier, within the range of the tier threshold.

[0035] The row threshold value is set in advance by the row threshold value setting unit 16.

[0036] (S16) S16 is a step for determining whether or not row numbers have been set for all boxes. If "row numbers have been set for all boxes" is not true, this means that there are boxes remaining for which row numbers have not been set. If "row numbers have been set for all boxes" is not true, the process proceeds to S17. From S17 onwards, row numbers are set for boxes for which row numbers have not been set.

[0037] On the other hand, if row numbers have been set for all the boxes, the process proceeds to step S18. After step S18, the boxes are removed.

[0038] First, we will explain the case where "row numbers have been set for all boxes" is not true. (S17) S17 is a step for incrementing the value of n by 1. In this case, the value of n is incremented from 1 to n=2. Then, the process proceeds to S15 again. This time, in S15, n=2 and boxes in the second row are set. Then, the process proceeds to S16 again, and the same processing is repeated in S17 and S15 until row numbers have been set for all boxes.

[0039] Once the step numbers have been set for all the boxes, the process proceeds to step S18. (S18) S18 is a step for outputting the detection results including the step numbers of all the detected boxes. In S18, the step number output unit 15 outputs the step numbers set by the step setting unit 14 in a format that can be used by the program of the robot 40. The robot program can refer to the output step numbers within the program. This makes it possible to obtain a robot program that takes the step numbers into account.

[0040] (S19) S19 is a step in which a command is issued to the robot to pick up only boxes with a tier number of 1 (the top tier). This command causes the robot to pick up only the top tier box. The robot also picks up all of the top tier boxes.

[0041] (S11) When all boxes on the top row have been removed, the process returns to step S11. In S11, the image acquisition unit 11 again acquires a two-dimensional image and a distance image. Next, after detection in S12, the process proceeds to the determination step of S13. In S13, if the number of detections is one or more, there are still boxes that have not been removed. The process proceeds to step S14 again. For the remaining boxes, a row is set and they are removed.

[0042] In S13, if the number of detections is not 1 or more, there are no boxes remaining. The process proceeds to S20, and the flow ends.

[0043] (Number of times of imaging and detection) As shown in the flow described with reference to FIG. 3 , in the image processing device 1 of this embodiment, imaging by the image acquisition unit 11 and detection by the detection unit 13 are performed each time the topmost box (object 101) is removed. Conventionally, imaging and detection are performed each time one object 101 is removed. There are often two or more objects 101 on the topmost level. Therefore, in the image processing device 1 of this embodiment, it is possible to reduce the number of times of imaging and detection.

[0044] As described above, the image processing device 1 of this embodiment sets stages for picking up the objects 101. Therefore, it is possible to pick up an object 101 by capturing an image and detecting it each time it is picked up up to any number of stages. As a result, it is possible to pick up as many objects 101 as possible with nothing on them, with fewer image captures.

[0045] 2, in the image processing device 1 of this embodiment, steps are set downward from the top surface 102 of the object 101. Therefore, each step always includes at least one object 101. As a result, it is possible to prevent the number of steps from increasing unnecessarily.

[0046] For example, if the logic for extracting the object 101 is a logic for taking an image each time a stage is changed, preventing an unnecessary increase in the number of stages leads to preventing an unnecessary increase in the number of times images are taken.

[0047] (Robot Program) By using the number of stages of the object 101 output from the number of stages output unit 15 in the robot program, it becomes easier to create logic using the number of stages.

[0048] Furthermore, in the image processing device 1 of this embodiment, the number of stages in the top stage is always set to 1 regardless of the situation. Therefore, if it is desired to pick up only the object 101 in the top stage, the robot's picking program can be set to pick up only the object 101 with a stage number of 1. This makes it easy to set up logic for picking up the object.

[0049] (Multiple Layers) In the explanation with reference to FIG. 3 , an example was described in which imaging and detection are performed each time a box with one layer (the top layer) is removed. The frequency of imaging and detection is not limited to each time a box with only one layer is removed. Imaging and detection can also be performed each time boxes with one and two layers, i.e., two layers, are removed. For example, when there is no risk in removing two layers of boxes without taking an image or the like in between due to the shape of the object 101 or the way the object 101 is stacked, imaging and detection can be performed each time two layers of boxes are removed. This can further reduce the number of times imaging and detection are performed.

[0050] When imaging and detection are performed each time a box having multiple layers is taken out, the number of layers is not limited to two, and may be three or more.

[0051] (Embodiment 2) An image processing device 1 according to embodiment 2 will be described with reference to Fig. 4 to Fig. 7. The following description will focus on the differences between embodiment 2 and embodiment 1. Items not specifically described can be the same as those in embodiment 1.

[0052] (Step Setting) An example of step setting in the second embodiment will be described. The step setting in the second embodiment differs from the step setting in the first embodiment. This will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining the step setting, and is a view of a stack viewed from the side. The stack shown in FIG. 4 is referred to as stack 120. The objects 101 included in stack 120 are referred to as objects 121 to 125. In FIG. 4, as in FIG. 2, the positions of the objects in front and behind the paper surface are not shown. The order from object 121 to object 125 is from the object whose top surface 102 is located at the top to the object whose top surface 102 is located at the bottom.

[0053] In the second embodiment, the detected objects are divided downward in increments of a predetermined threshold value of one step height, starting from the top surface 102 of the object whose top surface 102 is located at the highest position, and the number of steps is set from the lowest step to the top. In the example shown in FIG. 4 , the detected object whose top surface 102 is located at the highest position is object 121. Therefore, the detected objects are divided downward in increments of a predetermined threshold value of one step height 311, starting from the top surface 102 of object 121. The step including the bottom edge 103 of the lowest object 121 is set as the lowest step. In the example shown in FIG. 4 , the bottom edges 103 of objects 124 and 125 are the lowest-positioned bottom edges 103. Therefore, the step including objects 124 and 125 is set as the lowest step.

[0054] Then, the number of stages is set from the lowest stage as the first stage to the top stage. In the example shown in Fig. 4, stages 1 to 5 are set in order from bottom to top 300. The stage to which an object belongs is the stage on which the upper surface 102 of the object is located. In the example shown in Fig. 4, objects 121 and 122 are in the fifth stage. Object 123 is in the fourth stage. Object 124 is in the third stage. Object 125 is in the second stage. There are no objects in the first stage.

[0055] The method of setting the tiers in the second embodiment described above is effective, for example, when detecting objects by capturing images of a wide range using a plurality of visual sensors 30. Before describing the order in which the objects 101 are taken out in the second embodiment, an overview of the depalletizing system 200 in the second embodiment will be described.

[0056] (Depalletizing system) Fig. 5 is a diagram showing a depalletizing system 200 of embodiment 2. As shown in Fig. 5, in the depalletizing system 200 of embodiment 2, two stacks are the targets of removal. The two stacks are a first stack 104 and a second stack 105. The objects 101 contained in the first stack 104 and the objects 101 contained in the second stack 105 are removed based on the stacks.

[0057] The depalletizing system 200 of the second embodiment includes two visual sensors. The two visual sensors are a first visual sensor 31 and a second visual sensor 32. The first visual sensor 31 is disposed above 300 the first stack 104. The second visual sensor 32 is disposed above 300 the second stack 105.

[0058] If all the objects (boxes) cannot fit within the field of view of a single visual sensor, multiple visual sensors can be used as shown in Fig. 5. As shown in Fig. 5, it is preferable to use multiple perception sensors when there are a large number of objects 101 and the objects 101 are arranged in multiple stacks.

[0059] In the second embodiment, the number of stacked bodies is two and the number of visual sensors is two. The number of stacked bodies is not limited to two and may be three or more. The number of visual sensors is not limited to two and may be three or more.

[0060] (Image Processing Apparatus) The image processing apparatus 1 of the second embodiment includes a take-out stage number control unit 17 and a take-out stage number setting unit 18 in addition to the units included in the image processing apparatus 1 of the first embodiment.

[0061] (Removal stage number control unit) The removal stage number control unit 17 is a unit that makes it possible to remove only the objects 101 of a desired stage number by deleting detection results other than the desired stage number among the stage number set by the stage setting unit 14, or by setting a flag that allows removal only for the desired stage number. For example, the removal stage number control unit 17 makes it possible to exclude a stage that does not contain the objects 101 from being removed.

[0062] (Removal Stage Number Setting Unit) The removal stage number setting unit 18 is a part that sets how many stages to remove.

[0063] In the image processing device 1 of the second embodiment, when an arbitrary number of layers of all objects (boxes) including object 1 contained in the first stack 104 and object 2 contained in the second stack 105 are removed from the top layer, imaging is performed by the visual sensor and detection is performed by the detection unit. The order in which the objects are removed will be specifically described below.

[0064] 6 is a diagram for explaining the removal sequence of the objects, and is a view of the stack from the side. The side direction is a direction perpendicular to the upper side 300, as in FIGS. 2 and 4. In the example shown in FIG. 6, the objects are stacked in two stacks, a first stack 130 and a second stack 140.

[0065] (Layers of the First Stack) The setting of the tiers is performed for each stack. The method for setting the tiers follows the method described with reference to FIG. 4 . Setting of the tiers for the first stack 130 will now be described. In the first stack 130, the object whose top surface 102 is located at the highest position among the detected objects is object 131. Therefore, the stack is divided downward from the top surface 102 of object 131 in increments of a predetermined threshold 312, which is the height of one tier. The tier containing the lowest bottom edge 103, i.e., the bottom edge 103 of object 136 and the bottom edge 103 of object 137, is designated as the lowest tier. Next, the number of tiers is set from the bottom tier as the first tier to the top tier. In the stack 130 shown in FIG. 6 , tiers 1 to 7 are set in order from bottom to top 300.

[0066] The tier to which an object belongs is the tier on which the top surface 102 of the object is located. In the example of stack 130 shown in Figure 6, objects 131 and 132 are on the seventh tier. Object 133 is on the sixth tier. Object 134 and object 135 are on the fifth tier. Object 136 and object 137 are on the third tier. There are no objects on the fourth, second, or first tier.

[0067] (Ran of Second Stack) Setting of the tiers of the second stack 140 will be described. Setting of the tiers of the second stack 140 also follows the method described with reference to FIG. 4 . In the second stack 140, the object whose top surface 102 is at the highest position among the detected objects is the object 141. Therefore, the object 141 is divided downward from the top surface 102 thereof by a predetermined threshold 313 of one tier height. Here, the threshold 313 of one tier height can be set to the same as the threshold 312 of one tier height of the first stack 130.

[0068] The row containing the lowest bottom edge 103, i.e., the bottom edge 103 of the object 144 and the bottom edge 103 of the object 145, is set as the lowest row. Next, the number of rows is set from the lowest row as the first row to the top row. In the stack 140 shown in FIG. 6 , rows 1 to 5 are set in order from bottom to top 300. The stack height of the second stack 140 is lower than the stack height of the stack 130. Therefore, the number of rows of the second stack 140 is fewer than the number of rows of the first stack 130.

[0069] The tier to which an object belongs is the tier on which the top surface 102 of the object is located. In the example of stack 140 shown in Figure 6, objects 141 and 142 are on the fifth tier. Object 143 is on the fourth tier. Object 144 is on the third tier. Object 145 is on the second tier. There is no object on the first tier.

[0070] As shown in FIG. 6 , when multiple visual sensors (cameras) capture images of a wide area to detect an object (box), removal can be performed from the highest tier, including all of the detection results from the multiple visual sensors. In the example of FIG. 6 , removal is performed from the highest tier, including the first stack 104 and the second stack 105. By removing the object in this manner, even if there are multiple detection results, the object 101 can be removed in descending order of the position of the top surface 102 of the object 101. A specific removal procedure will be described below. Note that, for convenience of explanation, the tier numbers in the following description are the tier numbers that were initially set. The tier numbers may be reassigned each time imaging and detection is performed.

[0071] Specifically, first, objects 131 and 132, which are the objects in the seventh layer, are removed from the first stack 130. After removal, images are captured and detection is performed. Next, object 133, which is the object in the sixth layer, is removed from the first stack 130. After removal, images are captured and detection is performed.

[0072] Next, the fifth layer of objects is extracted, and the fifth layer of objects is included in both the first stack 130 and the second stack 140. Therefore, after all of the fifth layer of objects included in the first stack 130 and the second stack 140 are extracted, imaging and detection are performed.

[0073] Specifically, objects 134 and 135, which are objects in the fifth layer included in the first stack 130, and objects 141 and 142, which are objects in the fifth layer included in the second stack 140, are removed. After removal, imaging and detection are performed.

[0074] The same procedure is followed for the fourth and subsequent rows. After the object in the bottom row containing the object is removed, the removal procedure ends.

[0075] By using the removal procedure described above, even if there are multiple detection results based on images captured by multiple visual sensors, it is possible to remove objects with as few images captured as possible in descending order of position, with as few objects as possible on top.

[0076] In the second embodiment, similarly to the first embodiment, imaging and detection can be performed not every time an object in one stage is taken out, but every time a plurality of stages are taken out.

[0077] The processing flow will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of picking up an object in the second embodiment. The following explanation will focus on points that are different from the flow in the first embodiment, which was previously explained with reference to Fig. 3. The flow starts from S20 and ends at S29.

[0078] (S20 to S23) S20 to S23 are the same as S10 to S13 in the first embodiment.

[0079] (S24) S24 is a step of dividing the boxes downwards by the [row threshold] from the top of the highest box. In S24, rows are divided downwards by the [row threshold] from the top of the box (object) with the highest top surface among the boxes (objects) for which no row number is set. When the row containing the bottom edge of the object with the lowest bottom edge is divided, the row division is completed.

[0080] The row threshold value is set in advance by the row threshold value setting unit 16.

[0081] (S25) S25 is a step for determining the range of the number of rows, with the lowest row being row 1. In S25, the number of rows is determined from bottom to top for the rows divided in S24. The lowest row is always row 1.

[0082] (S26) S26 is a step in which, for a box whose number of layers has not been set, the layer number on which the top surface of the box is located is set as the layer number of the box. In S26, the layer on which the top surface of the box is located is set as the layer of the box, among the layers separated in S25.

[0083] (S27) S27 is a step for determining whether or not row numbers have been set for all boxes. S27 is the same as S16 in the first embodiment. If "row numbers have been set for all boxes" is not true, this means that there are boxes remaining for which row numbers have not been set. If "row numbers have been set for all boxes" is not true, the process returns to S26. In S26, row numbers are set for boxes for which row numbers have not been set.

[0084] On the other hand, if row numbers have been set for all the boxes, the process proceeds to step S28, where the boxes are removed.

[0085] (S28) S28 is a step in which a command is issued to the robot to take out the boxes corresponding to the [number of layers to be taken out], starting with the box with the largest number of layers. If the [number of layers to be taken out] is one layer, a command is issued to the robot to take out the box from the layer with the largest number of layers (only one layer). In response to this command, the robot takes out only the box from the layer with the largest number of layers. The robot also takes out all of the boxes from the layer with the largest number of layers.

[0086] The number of take-out stages is set in advance by the take-out stage number control unit 17.

[0087] (S21) When all boxes in the row with the largest number of rows have been removed, the process returns to step S21. In S21, the image acquisition unit 11 again acquires a two-dimensional image and a distance image. Next, after detection in S22, the process proceeds to the judgment step of S23. In S23, if the number of detections is one or more, there are still boxes that have not been removed. The process proceeds to step S24 again. For the remaining boxes, rows are set and they are removed.

[0088] In S23, if the number of detections is not 1 or more, there are no boxes remaining. The process proceeds to S29, and the flow ends.

[0089] (Embodiment 3) An image processing device 1 according to embodiment 32 will be described with reference to Figs. 8 and 9. The following description will focus on the differences between embodiment 3 and embodiment 1. Items not specifically described can be the same as embodiment 1. Embodiment 3 differs from embodiment 1 in the way the number of stages is set.

[0090] (Depalletizing System) Fig. 8 is a diagram showing a depalletizing system 200 of embodiment 3. As shown in Fig. 3, the depalletizing system 200 of embodiment 3 includes a control unit 10, a visual sensor 33, and a robot 40, similar to embodiment 1.

[0091] (Image Processing Device) Unlike the image processing device 1 of the first embodiment, the image processing device 1 included in the control unit 10 of the third embodiment does not include a row threshold setting unit 16. This is because the image processing device 1 of the third embodiment sets the number of rows without using row thresholds.

[0092] (Setting of Tiers) In the third embodiment, the setting of tiers is performed by clustering the heights of the objects (boxes). By clustering, for example, it is meant grouping the objects 101 according to the proximity of the heights of the top surfaces 102. An example of a clustering method is Ward's method. Ward's method is a hierarchical clustering method that clusters data so as to minimize variance. Note that the clustering method is not limited to Ward's method.

[0093] Specifically, the visual sensor 33 captures images, and the objects 101 are grouped into stages based on the results of detection by the detection unit 13. The objects 101 are then removed for each group. This makes it possible to remove objects 101 with as few images as possible and with as little material on top as possible, as in the first embodiment.

[0094] An example of the processing flow will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of picking up an object in the third embodiment. The following explanation will focus on points that differ from the flow in the first embodiment, which was previously explained with reference to Fig. 3. The flow starts from S30 and ends at S38.

[0095] (S30 to S33) S30 to S33 are the same as S10 to S13 in the first embodiment.

[0096] (S34) S34 is a step of clustering the boxes by height. In S34, boxes (objects) whose top surfaces 102 are close in height are grouped. The grouping is performed by the stage setting unit 14.

[0097] (S35) S35 is a step for setting the number of layers for each box, with the number of layers decreasing as the cluster gets taller. The highest cluster, i.e., the top cluster, is set to the first layer. As the height of each cluster decreases, the number of layers increases accordingly. Once the number of layers has been set for all boxes, the process proceeds to step S36.

[0098] (S36 and S37) S36 and S37 are the same as S18 and S19 in the first embodiment, respectively. In S36, the tier number output unit 15 outputs the tier number set by the tier setting unit 14 in a format that can be used by the program of the robot 40. In addition, in S37, an instruction is given to the robot to pick up only boxes with a tier number of 1 (top tier). This instruction causes the robot to pick up only the box on the top tier. In addition, the robot picks up all of the boxes on the top tier.

[0099] (S31) When all boxes on the top row have been removed, the process returns to S31. In S31, the image acquisition unit 11 again acquires a two-dimensional image and a distance image. Next, after detection in S32, the process proceeds to the determination step of S33. In S33, if the number of detections is one or more, there are still boxes that have not been removed. The process proceeds to S34 again. The remaining boxes are clustered and removed.

[0100] In S33, if the number of detections is not 1 or more, there are no boxes remaining. The process proceeds to S38, and the flow ends.

[0101] In the third embodiment, rows are set without using row thresholds, which eliminates the need for a row threshold setting unit, thereby simplifying the configuration of the image processing device 1.

[0102] 9, an example has been described in which imaging and detection are performed each time a box with one tier (the top tier) is removed. In the third embodiment, imaging and detection can also be performed each time two or more tiers of boxes are removed, similar to the first embodiment.

[0103] The present disclosure is not limited to the above-described embodiments and modifications, and includes modifications and improvements within the scope of achieving the object of the present disclosure.

[0104] The following additional notes are further disclosed regarding the above embodiment.

[0105] (Supplementary Note 1) An image processing device (1) comprising: an image acquisition unit (11) that acquires image information including height information; a detection unit (13) that performs a detection process to detect an object from the image information based on information representing the characteristics of the object; and a tier setting unit (14) that sets, as the top tier for removal, an object whose top surface is located within a threshold range below the top surface of the object that is located highest among the objects detected by the detection unit (13).

[0106] (Note 2) In the image processing device (1) described above, the tier setting unit (14) sets the object that is located within a threshold range from the top surface of the highest object among the objects below the top tier as the second tier from the top, and thereafter repeats this process for each detected object to set the number of tiers.

[0107] (Supplementary Note 3) In the image processing device (1), the tier setting unit (14) divides the top surface of the object at the highest position downward by threshold values, and sets the number of tiers from the bottom tier to the top tier, with the bottom tier being the first tier.

[0108] (Note 4) The image processing device (1) includes a step number output unit (15) that outputs the step number set by the step setting unit (14) in a form that can be used by a robot program.

[0109] (Supplementary Note 5) The image processing device (1) described above includes a step threshold setting unit (16) that sets a step threshold that determines the height of one step in the step setting unit (14).

[0110] (Note 6) The image processing device (1) described above includes a removal stage number control unit (17) that removes only the target object of the desired stage number by deleting detection results other than the desired stage number in the stage number set by the stage setting unit (14) or by setting a flag indicating that only the desired stage number can be removed.

[0111] (Supplementary Note 7) In the image processing device (1), the take-out stage number control unit (17) includes a take-out stage number setting unit (18) that sets how many stages to take out.

[0112] (Supplementary Note 8) In the image processing device (1), the row setting unit (14) sets the number of rows based on the row height instead of a threshold value, or on clusters clustered in a three-dimensional point cloud.

[0113] REFERENCE SIGNS LIST 1 Image processing device 10 Control unit 11 Image acquisition unit 12 Height acquisition unit 13 Detection unit 14 Layer setting unit 15 Layer number output unit 16 Layer threshold setting unit 17 Pick-up layer number control unit 18 Pick-up layer number setting unit 30 Visual sensor 40 Robot 100 Stacked body 101 Object 102 Upper surface 200 Depalletizing system 300 Above

Claims

1. An image processing apparatus comprising: an image acquisition unit that acquires image information including height information; a detection unit that performs a detection process of detecting the object from the image information based on information representing characteristics of the object; and a stage setting unit that sets, as the uppermost stage for extraction, an object whose upper surface is located within a threshold range downward from the upper surface of the object located at the highest position among the objects detected by the detection unit.

2. The image processing apparatus according to claim 1, wherein the stage setting unit sets, as the second stage from the top, an object whose upper surface is within the threshold range from the upper surface of the object located at the highest position among the objects below the uppermost stage, and thereafter, repeats this process for the detected objects to set the number of stages.

3. The image processing apparatus according to claim 1, wherein the stage setting unit divides downward by the threshold value from the upper surface of the object located at the highest position, and sets the number of stages from the lowermost stage as the first stage to the uppermost stage.

4. The image processing apparatus according to claim 1, further comprising a stage number output unit that outputs the number of stages set by the stage setting unit in a form that can be used in the robot program.

5. The image processing apparatus according to claim 3, further comprising a stage threshold setting unit that sets a threshold value for a stage that defines the height of one stage by the stage setting unit.

6. The image processing apparatus according to claim 3, comprising an extraction stage control unit that extracts only objects of an arbitrary number of stages by deleting detection results other than the arbitrary number of stages or setting a flag that allows only the objects of the arbitrary number of stages to be extracted at the number of stages set by the stage setting unit.

7. The image processing apparatus according to claim 6, wherein the extraction stage control unit comprises an extraction stage setting unit that sets the number of stages to be extracted.

8. The image processing apparatus according to claim 1, wherein the stage setting unit sets the number of stages by the height of the stage or by clusters obtained by clustering in a three-dimensional point cloud instead of the threshold value.

Images