Image processing device, part gripping system, image processing method, and part gripping method

Abstract

A composite image Ic is generated by composing a grayscale image Ig (luminance image) and a depth image Id representing each of a plurality of tools P (step S201). The composite image Ic thus composed, in which a shape of a tool P in a relatively high position among the plurality of tools P is likely to remain, is useful for recognizing such a tool P.

Description

[Technical field]

[0001] The present invention relates to a technique for gripping a plurality of parts stored in a container by a robot hand, and is particularly applicable to bin picking. [Background technology]

[0002] In Patent Document 1, when a robot hand grasps randomly stacked parts, an image of the parts is used to recognize the parts to be grasped. Specifically, an area having a brightness equal to or higher than a predetermined threshold is identified from the image of the piled parts, and point cloud data is generated for this area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-147256 A Summary of the Invention [Problem to be solved by the invention]

[0004] In bin picking, in order to reliably grasp parts, it is preferable to target the parts that are closest to the robot hand, i.e., at a higher position, among the multiple piled parts. However, in the method of Patent Document 1, point cloud data is generated by limiting the area of ​​the captured image that has a brightness equal to or higher than a threshold. Therefore, if the brightness of the captured image does not accurately reflect the height of the parts, the generated point cloud data is not necessarily useful for recognizing the parts at a higher position.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique that is useful for recognizing a part that is located in a relatively high position among a plurality of parts stored in a container. [Means for solving the problem]

[0006] The image processing device of the present invention includes an image acquisition unit that acquires a brightness image showing multiple parts stored in a container and a depth image showing the multiple parts, and an image synthesis unit that generates a synthetic image by synthesizing the brightness image and depth image acquired by the image acquisition unit.

[0007] The image processing method of the present invention includes a step of acquiring a brightness image showing a plurality of parts stored in a container and a depth image showing the plurality of parts, and a step of generating a composite image by combining the brightness image and the depth image.

[0008] In the image processing device and method configured in this manner, a composite image is generated by combining brightness images and depth images each showing a plurality of parts. In the composite image generated in this manner, the shapes of parts at relatively high positions among the plurality of parts tend to remain, which is useful for recognizing such parts.

[0009] The image processing device may be configured such that the luminance image indicates the luminance Vg for each pixel, the depth image indicates the depth Vd for each pixel, and the composite image indicates the composite value Vc for each pixel, and the image composition unit generates the composite image by calculating the composite value Vc in which the luminance Vg is weighted by the depth Vd. The composite image generated in this manner is likely to retain the shapes of components at relatively high positions among multiple components, and is useful for recognizing such components.

[0010] In addition, the image synthesis unit is Vc = Vd × (1 + Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the luminance image The image processing device may be configured to generate a composite image based on the above-mentioned. The composite image generated in this manner is likely to retain the shapes of components that are located at relatively high positions among a plurality of components, and is useful for recognizing such components.

[0011] The image processing device may be further configured to include a success probability calculation unit that generates a plurality of target part images each showing a different part by setting a hand range indicating the range of the robot hand for the part and creating a target part image showing the part included in the hand range based on the composite image, and calculates the grasping success probability for each of the plurality of target part images when the robot hand attempts to grasp the part in the positional relationship between the robot hand and the part shown in the target part image. In this configuration, a hand range indicating the range of the robot hand is set for the part included in the composite image, a target part image showing the part included in the hand range is created based on the composite image, and a grasping success probability is calculated for each target part image. As described above, the composite image used at this time tends to retain the shape of the part at a relatively high position. Therefore, it is possible to reliably generate a target part image including a part at a relatively high position, in other words, a part with a relatively high grasping success probability.

[0012] The image processing device may be configured so that the success probability calculation unit selects one target part image from among the plurality of target part images based on the grasping success probability calculated for each of the plurality of target part images, and determines that the part indicated by the one target part image is to be grasped by the robot hand in the posture indicated by the one target part image. With such a configuration, it is possible to reduce the frequency of failure to grasp the part by the robot hand.

[0013] In addition, the image processing device may be configured such that the success probability calculation unit selects, as one target part image, a target part image having the highest gripping success probability from among a plurality of target part images that do not satisfy a predetermined exclusion condition. With such a configuration, it is possible to reliably reduce the frequency of failure in gripping a part by the robot hand.

[0014] For example, the predetermined exclusion condition is that the distance between the edge of the container and the target part image is less than a predetermined value, which can prevent the occurrence of a situation in which gripping of the part fails due to the influence of the wall surface of the edge of the container.

[0015] The image processing device may be configured so that the success probability calculation unit has a neural network that has learned the relationship between the target part image and the probability of successful grasping using training data on the success or failure of grasping when the robot hand attempts to grasp a part in the positional relationship between the robot hand and the part shown in the target part image, and the neural network calculates the probability of successful grasping for each of the multiple target part images. With this configuration, it is possible to reliably reduce the frequency of failure in grasping a part by the robot hand by using the learning results of the neural network.

[0016] Furthermore, the present invention is suitable for learning by such a neural network. That is, as an example of a learning method of a neural network, it is possible to input both a brightness image and a depth image to the neural network and have the neural network learn the relationship between these images and the grasping success probability. However, it is difficult for the neural network to learn the grasping success probability of the robot hand from the association between the brightness of the part shown in the brightness image and the height of the part shown in the depth image, and the calculation load is large. On the other hand, in the present invention, a composite image obtained by combining the brightness image and the depth image is input to the neural network, so that the neural network can learn based on this composite image. As a result, learning can be performed relatively easily and the calculation load can be reduced.

[0017] The image processing device may be configured to further include a success / failure determination unit that acquires gripping state information indicating a state in which a part indicated by a target part image is gripped by a robot hand having a posture indicated by the target part image, and determines whether the robot hand has successfully gripped the part based on the gripping state information. With such a configuration, it is possible to confirm whether the selected part has been successfully gripped based on the gripping success probability calculated for the target part image.

[0018] Various specific contents of the gripping state information can be assumed. For example, the success / failure determination unit can acquire, as the gripping state information, the mass measured by a mass meter when the part gripped by the robot hand from the container is placed on the mass meter. Alternatively, the success / failure determination unit can acquire, as the gripping state information, an image of the part gripped by the robot hand captured by a camera. In the latter case, the success / failure determination unit may acquire, as the gripping state information, multiple images of the part gripped by the robot hand captured by a camera from different directions.

[0019] The image processing device may be configured so that the success probability calculation unit causes the neural network to perform re-learning based on the result of acquiring the grip state information for the part to be gripped by the robot hand. In this configuration, the neural network is re-learned according to the result of the grip success of the part selected based on the grip success probability calculated for the target part image, thereby improving the accuracy of the calculation of the grip success probability by the neural network.

[0020] The part grasping system of the present invention comprises the above-mentioned image processing device, and a robot hand that grasps a part based on a decision by the image processing device to select one target part image from a plurality of target part images and grasp the part indicated by the one target part image with the robot hand in the posture indicated by the one target part image.

[0021] The part grasping method according to the present invention includes a step of acquiring a brightness image showing a plurality of parts stored in a container and a depth image showing the plurality of parts, a step of generating a composite image by combining the brightness image and the depth image, a step of generating a plurality of target part images each showing a different part by setting a hand range indicating the range of the robot hand for the parts and creating a target part image indicating the parts included in the hand range based on the composite image, and a step of calculating, for each of the plurality of target part images, a grasping success probability when the robot hand attempts to grasp the part in the positional relationship between the robot hand and the part indicated by the target part image, a step of selecting one target part image from among the plurality of target part images based on the grasping success probability calculated for each of the plurality of target part images, and determining that the part indicated by the one target part image will be grasped by the robot hand in the posture indicated by the one target part image, and a step of selecting one target part image from among the plurality of target part images, and grasping the part with the robot hand based on the determination that the part indicated by the one target part image will be grasped by the robot hand in the posture indicated by the one target part image.

[0022] In the part grasping system and method configured in this manner, a composite image is generated by combining brightness images and depth images showing each of a plurality of parts. The composite image thus generated tends to retain the shapes of parts at relatively high positions among the plurality of parts, and is useful for recognizing such parts. Furthermore, the robot hand grasps the parts based on the results of the part recognition in this manner. Therefore, it is possible to grasp the parts with the robot hand with a high success rate. Effect of the Invention

[0023] According to the present invention, it is possible to obtain an image (a composite image) that is useful for recognizing a part that is located at a relatively high position among a plurality of parts stored in a container. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a plan view illustrating an example of a part gripping system according to the present invention. [Diagram 2]FIG. 2 is a perspective view showing a schematic diagram of a robot hand used for gripping a part in the part gripping system shown in FIG. 1. [Diagram 3] FIG. 2 is a block diagram showing an example of an electrical configuration of a control device. [Figure 4A] 4 is a flowchart showing an example of bin picking executed by the part gripping system of FIG. 1. [Figure 4B] 4B is a flowchart showing an example of patch image processing executed in the bin picking of FIG. 4A. [Figure 4C] 4B is a flowchart showing an example of grasping inference performed in the bin picking of FIG. 4A. [Figure 4D] 4D is a flowchart showing an example of determination of a part to be grasped, which is executed in the grasp inference of FIG. 4C. [Figure 5A] FIG. 4C is a diagram showing a schematic diagram of operations performed in the patch image processing of FIG. 4B. [Figure 5B] FIG. 4C is a diagram showing a schematic diagram of operations performed in the patch image processing of FIG. 4B. [Figure 5C] FIG. 4C is a diagram showing a schematic diagram of operations performed in the patch image processing of FIG. 4B. [Figure 5D] FIG. 4C is a diagram showing a schematic diagram of operations performed in the patch image processing of FIG. 4B. [Figure 5E] FIG. 4C is a diagram showing a schematic diagram of operations performed in the patch image processing of FIG. 4B. [Figure 6A] FIG. 4D is a diagram showing a schematic diagram of an operation executed in the grasp inference of FIG. 4C. [Figure 6B] FIG. 4D is a diagram showing a schematic diagram of an operation executed in the grasp inference of FIG. 4C. [Figure 6C] FIG. 4D is a diagram showing a schematic diagram of an operation executed in the grasp inference of FIG. 4C. [Figure 7] FIG. 4D is a diagram showing a schematic diagram of an operation executed in the grasp inference of FIG. 4C. [Figure 8A] 11 is a flowchart showing an example of a method for collecting training data for an alignment neural network. [Figure 8B] FIG. 10 is a diagram illustrating an example of a position determination mask generated from a patch image. [Figure 9A]8B is an example of a flowchart for training an alignment neural network on the training data collected in FIG. 8A. [Figure 9B] FIG. 13 is a diagram illustrating an example in which the use of a mask is advantageous for calculating a loss function. [Figure 10A] 1 is an example of a flowchart for training a grasp classification neural network. [Figure 10B] 1 is an example of a flowchart for training a grasp classification neural network. [Figure 10C] 1 is an example of a flowchart for training a grasp classification neural network. [Figure 11] 10 is a flowchart showing an example of a method for re-learning the grasp classification neural network of the grasp classification network unit. [Figure 12] 13 is a modified example of the grasp classification neural network of the grasp classification network section. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Fig. 1 is a plan view showing an example of a part gripping system according to the present invention, and Fig. 2 is a perspective view showing a robot hand used to grip a part in the part gripping system of Fig. 1. In these figures and the following figures, the horizontal X direction, the horizontal Y direction perpendicular to the X direction, and the vertical Z direction are shown as appropriate. These X, Y, and Z directions form a global coordinate system. As shown in Fig. 1, the part gripping system 1 includes a control device 3 and a work robot 5, and the work robot 5 performs a task (bin picking) under the control of the control device 3.

[0026] Specifically, a parts bin 91 and a kitting tray 92 are arranged in the work space of the work robot 5. The parts bin 91 has a plurality of storage compartments 911 for storing parts, and a large number of parts are piled up in each storage compartment 911. The kitting tray 92 has a plurality of storage compartments 921 for storing parts, and a predetermined number of parts are placed in each storage compartment 921. The work robot 5 then picks up a part from the storage compartment 911 of the parts bin 91 (bin picking) and transfers it to the storage compartment 921 of the kitting tray 92. In addition, a trash can 93 is arranged between the parts bin 91 and the kitting tray 92, and when a defective part is detected, the work robot 5 discards the defective part in the trash can 93.

[0027] The working robot 5 is a scalar robot equipped with a robot hand 51 provided at the tip, and by gripping a part with the robot hand 51 and moving the robot hand 51, the robot hand 51 transfers the part from a part bin 91 to a kitting tray 92 and disposes of the part in a trash can 93. As shown in FIG. 2, the robot hand 51 has degrees of freedom in the X direction, the Y direction, the Z direction, and the θ direction. Here, the θ direction is a rotation direction centered on a rotation axis parallel to the Z direction. The robot hand 51 also has two claws 511 arranged in the gripping direction G, and each claw 511 has a flat plate shape perpendicular to the gripping direction G. The robot hand 51 can expand and reduce the distance between the two claws 511 in the gripping direction G, and grips the part by pinching it from the gripping direction G with these claws 511. Note that in FIG. 2, the gripping direction G is parallel to the X direction, but depending on the position of the robot hand 51 in the θ direction, the gripping direction G can be tilted with respect to the X direction as a matter of course.

[0028] Furthermore, the part holding system 1 includes two cameras 81 and 83, and a mass meter 85. The camera 81 is a plan view camera that captures images of a large number of parts piled up in the compartment storage 911 of the part bin 91 from the Z direction (above), and faces the working space of the working robot 5 from the Z direction. This camera 81 captures a grayscale image (two-dimensional image) showing the image target (part) in terms of brightness, and a depth image (three-dimensional image) showing the distance to the image target. As a specific method for acquiring the depth image, a phase shift method or a stereo matching method can be used. The camera 83 is a side view camera that captures images of the part held by the robot hand 51 from the Y direction, and is attached horizontally to the base of the robot hand 51. This camera 83 captures a grayscale image (two-dimensional image) showing the image target (part) in terms of brightness. In addition, the mass meter 85 measures the mass of the part placed in the compartment storage 921 of the kitting tray 92.

[0029] FIG. 3 is a block diagram showing an example of an electrical configuration of the control device. The control device 3 is, for example, a personal computer equipped with a calculation unit 31, a storage unit 35, and a UI (User Interface) 39. The calculation unit 31 is, for example, a processor equipped with a CPU (Central Processing Unit) or the like, and has a main control unit 311 and an image processing unit 4. The main control unit 311 and the image processing unit 4 are deployed in the calculation unit 31 by executing a predetermined program. The main control unit 311 executes control of hardware such as the robot hand 51, the cameras 81 and 83, and the mass meter 85 described above, and the image processing unit 4 executes image processing for recognizing a part to be grasped by the robot hand 51. In particular, the image processing unit 4 has an image synthesis unit 41, a patch image generation unit 43, a positioning network unit 45, and a grasp classification network unit 47. These functions will be described in detail later.

[0030] The storage unit 35 is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores, for example, programs and data for deploying the main control unit 311 or the image processing unit 4 in the calculation unit 31. The UI 39 has input devices such as a keyboard and a mouse, and an output device such as a display, and transfers information input by an operator via the input devices to the calculation unit 31 or the UI 39, and displays images on the display in response to commands from the calculation unit 31.

[0031] Figure 4A is a flowchart showing an example of bin picking performed in the part grasping system of Figure 1, Figure 4B is a flowchart showing an example of patch image processing performed in the bin picking of Figure 4A, Figure 4C is a flowchart showing an example of grasping inference performed in the bin picking of Figure 4A, and Figure 4D is a flowchart showing an example of determining a part to be grasped performed in the grasping inference of Figure 4C.

[0032] 4A, a plan view image of a large number of parts piled up in the storage compartment 911 of the part bin 91 is captured by the camera 81. As described above, a grayscale image Ig and a depth image Id are captured as this plan view image. The main control unit 311 transfers these images Id and Ig acquired from the camera 81 to the image synthesis unit 41 of the image processing unit 4, and the image synthesis unit 41 executes patch image processing (step S102).

[0033] Figures 5A to 5E are diagrams illustrating operations executed in the patch image processing of Figure 4B. In step S201 of the patch image processing of Figure 4B, the image synthesis unit 41 generates a synthetic image Ic (Figure 5C) by synthesizing a grayscale image Ig (Figure 5A) and a depth image Id (Figure 5B).

[0034] As shown in Fig. 5A, the grayscale image Ig is composed of a plurality of pixels PX arranged two-dimensionally in the X and Y directions, and is image data indicating the luminance Vg of the pixel PX for each of the plurality of pixels PX. Note that in Fig. 5A, a notation is used in which one pixel PX is specified by a combination (m,n) of "m" indicating a row number and "n" indicating a column number, and a pixel PX(m,n) of the grayscale image Ig has a luminance Vg(m,n). Note that the luminance Vg(m,n) has a larger value as the corresponding portion becomes brighter.

[0035] As shown in Fig. 5B, the depth image Id is composed of a plurality of pixels PX, similar to the grayscale image Ig, and is image data showing the depth (distance) of each of the plurality of pixels PX. In Fig. 5B, the same notation as in Fig. 5A is used, and a pixel PX(m,n) of the depth image Id has a depth Vd(m,n). Note that the depth Vd(m,n) has a larger value as the depth of the corresponding portion is shallower (in other words, the higher the position of the opposing portion).

[0036] As shown in Fig. 5C, the composite image Ic is composed of a plurality of pixels PX, similar to the grayscale image Ig, and is image data showing the composite value Vc of each of the plurality of pixels PX. In Fig. 5C, the same notation as in Fig. 5A is used, and the pixel PX(m,n) of the composite image Ic has a composite value Vc(m,n).

[0037] The composite value Vc(m,n) is expressed by the following formula: Vc(m,n)=Vd(m,n)×(1+Vg(m,n) / max(Vg)) max(Vg) is the maximum luminance Vg contained in the grayscale image Ig That is, the composite value Vc is the brightness Vg weighted by the depth Vd, and the composite image Ic is a depth-weighted grayscale image. In the above formula, the brightness Vg normalized by the maximum brightness is multiplied by the depth Vd (weight). However, normalization is not essential, and the composite value Vc may be calculated by multiplying the brightness Vg as it is by the depth Vd (weight). In short, it is sufficient to determine the composite value Vc so that it depends on both the brightness Vg and the depth Vd.

[0038] FIG. 5D shows an experimental result of generating a composite image Ic from a grayscale image Ig and a depth image Id. The grayscale image Ig (before filtering) is two-dimensional image data acquired by the camera 81, and the grayscale image Ig (after filtering) is two-dimensional image data obtained by removing a predetermined component (high-frequency component) from the two-dimensional image data acquired by the camera 81 using a filter. The depth image Id (before filtering) is three-dimensional image data acquired by the camera 81, and the depth image Id (before filtering) is three-dimensional image data obtained by removing a predetermined component (high-frequency component) from the three-dimensional image data acquired by the camera 81 using a filter. The composite image Ic is a depth-weighted grayscale image obtained by synthesizing the filtered grayscale image Ig and the depth image Id using the above formula. Here, when attention is paid to the ranges (elliptical ranges) designated by arrows in the columns of "grayscale image Ig (after filtering)" and "composite image Ic," the parts that were clearly shown in the grayscale image Ig (after filtering) are not shown in the composite image Ic. This is because the part in question is deep (in other words, low in height) and a small weight is applied to the luminance Vg of the part in question. In this way, the composition of the grayscale image Ig and the depth image Id has the effect of highlighting the part at a high position. Note that the filter used in FIG. 5D is not essential, and the same effect can be obtained even if it is omitted as appropriate.

[0039] The composite image Ic generated in step S201 of FIG. 4B is output from the image synthesis unit 41 to the patch image generation unit 43, and the patch image generation unit 43 executes image processing of steps S202 to S204 on the composite image Ic. Specific contents of this image processing are illustrated in FIG. 5E. In step S202, the composite image Ic is binarized using a predetermined threshold value to obtain a binary composite image Ic. In this binary composite image Ic, closed regions with high luminance (white) appear corresponding to parts, in other words, the closed regions in the binary composite image Ic can be recognized as parts P. In step S203, the patch image generation unit 43 executes labeling to associate different labels (numbers) with each part P (closed region Rc) in the binary composite image Ic.

[0040] In step S204, a cutout range Rc is set for cutting out an image including the part P from the binary composite image Ic. In particular, the cutout range Rc is set so as to represent the position of the robot hand 51 when gripping the part P. This cutout range Rc corresponds to the range (grasping target range) that the robot hand 51 targets for gripping, and the robot hand 51 can grip the part P present in the cutout range Rc. For example, in the "patch image Ip" column in FIG. 5E, the portion corresponding to the two claws 511 of the robot hand 51 facing the part P(2) from above in order to grip the part P is represented by a white solid line (parallel to the Y direction) in the cutout range Rc, and the trajectory of the movement of both ends of each claw 511 is represented by a white dashed line (parallel to the X direction). As can be seen from this example, the claws 511 are parallel to the Y direction, and the rotation angle of the robot hand 51 in the θ direction is zero degrees. In other words, the cutout range Rc is set in a state in which the rotation angle of the robot hand 51 in the θ direction is zero degrees. Then, the patch image generating unit 43 acquires an image of the cutout range Rc from the binary composite image Ic as a patch image Ip (patch image generation). This patch image Ip is generated for each part P labeled in step S203.

[0041] As shown in FIG. 4A, when the patch image processing in step S102 is completed, the grip inference in step S103 (FIG. 4C) is executed. FIGS. 6A to 6C and 7 are diagrams that diagrammatically show the operation executed in the grip inference in FIG. 4C. When the grip inference in FIG. 4C is started, patch image information (FIG. 6A) indicating a plurality of patch images Ip acquired in the patch image processing in step S102 is output from the image synthesis unit 41 to the alignment network unit 45. As shown in FIG. 6A, the patch image information indicates the patch image Ip, the label number of the patch image Ip, and the position of the cutout range Rc of the patch image Ip in association with each other. The shape of the cutout range Rc is the same for each patch image Ip, and the position (cutout position) of the cutout range Rc is specified by the X coordinate, Y coordinate, and θ coordinate of the geometric center of gravity of the cutout range Rc.

[0042] On the other hand, in step S301 of FIG. 4C, the alignment network unit 45 resets a count value for counting the labels of the multiple patch images Ip indicated by the patch image information to zero (step S301), and increments the count value (step S302).

[0043] In step S303, the alignment network unit 45 judges whether the area of ​​the object (white closed region) included in the patch image Ip of the current count value is appropriate. Specifically, the object area is compared with a lower threshold and an upper threshold that is greater than the lower threshold. If the object area is smaller than the lower threshold or greater than the upper threshold, the object area is judged to be inappropriate ("NO" in step S303), and the process returns to step S302. On the other hand, if the object area is equal to or greater than the lower threshold and equal to or less than the upper threshold, the object area is judged to be appropriate ("YES" in step S303), and the process proceeds to step S304.

[0044] In step S304, the alignment network unit 45 calculates a correction amount for correcting the position of the cut-out range Rc based on the patch image Ip of the current count value. That is, the alignment network unit 45 has an alignment neural network, which outputs the correction amount (Δx, Δy, Δθ) of the cut-out range Rc when the patch image Ip is input. The relationship between the patch image Ip and the correction amount of the cut-out range Rc will be described with reference to FIG. 6C.

[0045] In the "Cut-out range Rc" column of FIG. 6C, the cut-out range Rc and the patch image Ip cut out from the cut-out range Rc are shown, and in the "Corrected cut-out range Rcc" column, the corrected cut-out range Rcc obtained by correcting the position of the cut-out range Rc according to the correction amount (Δx, Δy, Δθ) is shown superimposed on the cut-out range Rc and the corrected cut-out range Rcc. The cut-out range Rc and the corrected cut-out range Rcc have the same shape, and the following operations are performed: · Parallel movement in the X direction by a correction distance Δx... X direction parallel operation · Parallel movement in the Y direction by the correction distance Δy...Y direction parallel operation Rotation in the θ direction by the correction angle Δθ...Rotation in the θ direction The cut-out range Rc after the above is executed coincides with the corrected cut-out range Rcc. Moreover, the positional deviation between the center of the corrected cut-out range Rcc and the part P is improved compared to the positional deviation between the center of the cut-out range Rc and the part P. In other words, the correction of the cut-out range Rc is a correction to improve the positional deviation between the cut-out range Rc and the part P, and more specifically, a correction to convert the cut-out range Rc into the corrected cut-out range Rcc so that the part P is centered. The alignment neural network of the alignment network unit 45 outputs a correction amount (Δx, Δy, Δθ) for correcting the cut-out range Rc of the patch image Ip in response to the input of the patch image Ip and calculating the corrected cut-out range Rcc. Incidentally, the calculation to convert the cut-out range Rc into the corrected cut-out range Rcc by the correction amount can be performed by multiplying a rotation matrix that rotates the cut-out range Rc by Δθ in the θ direction by a translation matrix that translates the cut-out range Rc in the X direction by Δx and translates the cut-out range Rc in the Y direction by Δy. Moreover, if it is necessary to take into account the enlargement or reduction of the image, a scaling matrix may be further multiplied.

[0046] 6C, when the part P has a shape that is long in a predetermined direction, it is preferable to perform centering so that the long axis direction of the part P is perpendicular to the gripping direction G of the robot hand 51. This enables the part P to be gripped accurately by the robot hand 51.

[0047] In step S305, the alignment network unit 45 creates a corrected cut-out range Rcc by correcting the cut-out range Rc based on the correction amount output by the alignment neural network, and obtains the image of the corrected cut-out range Rcc from the binary composite image Ic as a corrected patch image Ipc (corrected patch image generation). Then, steps S302 to S305 are repeated until steps S302 to S305 are completed for all labels included in the patch image information (in other words, all patch images Ip) (until step S306 becomes "YES").

[0048] When the process is completed for all labels, correction patch image information (FIG. 6B) indicating a plurality of correction patch images Ipc is output from the positioning network unit 45 to the grip classification network unit 47. As shown in FIG. 6B, the correction patch image information indicates the correction patch image Ipc, the label number of the correction patch image Ipc, and the position of the correction cut-out range Rcc of the correction patch image Ipc in association with each other. The shape of the correction cut-out range Rcc is the same for each correction patch image Ipc, and the position (cut-out position) of the correction cut-out range Rcc is specified by the X-coordinate, Y-coordinate, and θ-coordinate of the geometric center of gravity of the correction cut-out range Rcc.

[0049] In step S307, the grip classification network unit 47 calculates a grip success probability for each of the multiple corrected patch images Ipc shown in the corrected patch image information. Specifically, the grip success probability (grip success probability) is calculated when the robot hand 51 is positioned at the position (x+Δx, y+Δy, θ+Δθ) of the corrected cut-out range Rcc and the part P shown by the corrected patch image Ipc cut out in the corrected cut-out range Rcc is attempted to be gripped. That is, the grip classification network unit 47 has a grip classification neural network, and when the corrected patch image Ipc is input, this grip classification neural network outputs a grip success probability corresponding to the corrected patch image Ipc. In this way, the grip success probability information shown in FIG. 7 is acquired. As shown in FIG. 7, the grip success probability information indicates the corrected patch image Ip, the label number of the corrected patch image Ipc, the position of the corrected cut-out range Rcc of the corrected patch image Ipc, and the grip success probability of the corrected patch image Ipc in association with each other. Although the probability of successful grasping is indicated by a value between 0 and 1 in the example of FIG. 7, it may be indicated by a percentage.

[0050] In step S308, the main control unit 311 determines the part P to be grasped based on the grasping success probability information output from the grasping classification network unit 47. In determining the part to be grasped in Fig. 4D, each correction patch image Ipc of the grasping success probability information is sorted in descending order according to the grasping success probability (step S401). In other words, the correction patch image Ipc with a higher grasping success probability is sorted to a higher position.

[0051] In addition, correction patch images Ipc with the same grasping success probability are sorted in descending order according to the object area included in the correction patch image Ipc. That is, the correction patch image Ipc with a larger object area is sorted to a higher position. In step S403, the count value of the sorting order is reset to zero, and in step S404, the count value is incremented.

[0052] In step S405, it is determined whether the part P included in the correction patch image Ipc of the current count value is close to the edge of the storage compartment 911 (container) of the part bin 91. Specifically, if the distance between the position of the correction cut-out range Rcc from which the correction patch image Ipc is cut out and the wall surface of the storage compartment 911 is less than a predetermined value, it is determined that the part P is close to the edge of the container ("YES" in step S405), and the process returns to step S404. On the other hand, if the distance is equal to or greater than a predetermined value, it is determined that the part P is not close to the edge of the container ("NO" in step S405), and the process proceeds to step S406. In step S406, the correction patch image Ipc of the current count value is selected as one correction patch image Ipc indicating the part P to be grasped. Then, the process returns to the flowchart of FIG. 4A.

[0053] 4A, the robot hand 51 moves to a position indicated by the correction cut-out range Rcc corresponding to one correction patch image Ipc selected in step S103, and grasps the part P indicated by one correction patch image Ipc. In step S105, an image of the part P grasped by the robot hand 51 is captured by the camera 83, and in step S106, the main control unit 311 determines the part P grasped by the robot hand 51 from the image captured by the camera 83. Furthermore, the main control unit 311 determines whether the number of grasped parts P is one (step S107), and if the number is not one (if "NO" in step S107), causes the robot hand 51 to return the part P to the storage compartment 911 of the part bin 91 (step S108). Furthermore, if the number of grasped parts P is one (if "YES" in step S107), the main control unit 311 determines whether the grasped part P is normal (step S109), and if there is an abnormality such as the area representing the part P being too small (if "NO" in step S109), it causes the robot hand 51 to discard the part P in the trash can 93 (step S110).

[0054] On the other hand, if the part P is normal (if "YES" in step S109), the main control unit 311 causes the robot hand 51 to place the part P in the storage compartment 921 of the kitting tray 92 (step S111). Next, the main control unit 311 measures the mass using the mass meter 85 (step S112) and determines whether the mass indicated by the mass meter 85 is appropriate (step S113). Specifically, the determination can be made based on whether the mass corresponding to the part P placed on the kitting tray 92 has increased. Then, if the mass is not appropriate (if "NO" in step S113), the main control unit 311 notifies the operator of the abnormality using the UI 39, whereas if the mass is appropriate (if "YES" in step S113), the main control unit 311 returns to step S101.

[0055] The above is the content of bin picking executed by the part holding system 1. In the above-mentioned holding inference, the alignment network unit 45 calculates the correction amount (Δx, Δy, Δθ) for correcting the cut-out range Rc based on the patch image Ip cut out from the cut-out range Rc. In particular, the alignment network unit 45 calculates the correction amount of the cut-out range Rc from the patch image Ip using an alignment neural network. Next, a method for making the alignment neural network learn the relationship between the patch image Ip and the correction amount of the cut-out range Rc will be described.

[0056] 8A is a flowchart showing an example of a method for collecting learning data for an alignment neural network. This flowchart is executed by the calculation unit 31 of the control device 3. When executing this flowchart, a simulator that executes bin picking in a part holding system 1 virtually constructed by calculation (hereinafter, appropriately referred to as a "virtual part holding system 1") is configured in the calculation unit 31. This simulator virtually executes, by calculation, the operation of the robot hand 51 to hold a part P from a storage compartment 911 of a part bin 91, based on physical parameters such as gravitational acceleration and friction coefficient.

[0057] In step S501, it is confirmed whether the number of data necessary for learning has been acquired. This necessary number can be preset by, for example, an operator. If the necessary number of data has been acquired (YES in step S501), the flow chart of FIG. 8A ends, whereas if the number of acquired data is less than the necessary number (NO in step S501), the flow proceeds to step S502.

[0058] In step S502, it is determined whether or not a sufficient number of parts P are stored in the storage compartments 911 of the part bin 91 arranged in the virtual part holding system 1. Specifically, this can be determined based on whether the number of parts P is equal to or greater than a predetermined number. If the number of parts P in the storage compartments 911 of the part bin 91 is less than the predetermined number (if "NO" in step S502), the number of parts P in the storage compartments 911 of the part bin 91 is reset to an initial value (step S503), and the process returns to step S501. On the other hand, if the number of parts P in the storage compartments 911 of the part bin 91 is equal to or greater than the predetermined number (if "YES" in step S502), the process proceeds to step S504.

[0059] In step S504, a composite image Ic is generated in the virtual part holding system 1 in the same manner as in the above-described real part holding system 1. Next, the composite image Ic is binarized to generate a binary composite image Ic, and labeling is performed on each part P included in the binary composite image Ic (step S505). Then, a cutout range Rc is set for each labeled part P, and a patch image Ip is cut out (step S506).

[0060] In step S507, the count value for each patch image Ip is reset, and in step S508, the count value is incremented. Then, in the same manner as above, it is determined whether the area of ​​the object (white closed region) included in the patch image Ip of the current count value is appropriate (step S509). If the area of ​​the object is inappropriate (if "NO" in step S509), the process returns to step S508, whereas if the area of ​​the object is appropriate (if "YES" in step S509), the process proceeds to step S510.

[0061] When one patch image Ip having an appropriate area of ​​the object is selected in this way, the main control unit 311 generates a position determination mask Mp (FIG. 8B) from the one patch image Ip (step S510). FIG. 8B is a diagram showing an example of a position determination mask generated from a patch image. This position determination mask Mp has an outline of the same shape as the patch image Ip (in other words, the cutout range Rc), and a component reference pattern Pr of the same shape as the component P included in the patch image Ip is disposed in the center of the position determination mask Mp. This component reference pattern Pr is generated so as to have the number of pixels in each of the vertical and horizontal directions of the component P (in other words, the closed white region) included in the patch image Ip. This position determination mask Mp is a model of an ideal patch image Ip in which the component P is located at the center. Then, the patch image Ip and the position determination mask Mp generated from the patch image Ip are associated with each other and stored in a patch image list (step S511).

[0062] Once each step up to step S511 is completed in this manner, the process returns to step S501. Steps S501 to S511 are then repeatedly executed until a required number of data are acquired, in other words, until the number of pairs of patch images Ip and position determination masks Mp stored in the patch image list reaches a required number.

[0063] Fig. 9A is an example of a flowchart for making the alignment neural network learn the learning data collected in Fig. 8A. This flowchart is executed by the calculation unit 31 of the control device 3. In step S601, it is determined whether the number of learning operations has reached a predetermined number. This predetermined number can be set in advance by, for example, an operator.

[0064] In step S602, an unlearned patch image Ip selected from the patch image list is forward propagated to the alignment neural network of the alignment network unit 45. As a result, a correction amount (Δx, Δy, Δθ) corresponding to the patch image Ip is output from the neural network of the alignment network unit 45. Furthermore, the alignment network unit 45 generates a corrected patch image Ipc (step S603) by cutting out a binary composite image Ic (generated in step S505) within the corrected cut-out range Rcc obtained by correcting the cut-out range Rc by this correction amount.

[0065] In step S604, the alignment network unit 45 overlays the position determination mask Mp corresponding to the patch image Ip selected in step S602 and the corrected patch image Ipc so that their contours match, and calculates the mean square error between the component reference pattern Pr of the position determination mask Mp and the component P included in the corrected patch image Ipc as a loss function. Then, in step S605, the alignment neural network parameters are updated by backpropagating this loss function in the alignment neural network (error backpropagation).

[0066] The loss function can be calculated without using the position determination mask Mp. That is, the principal axis angle may be calculated from the moment of the image of the part P, and the mean square error between this principal axis angle and a predetermined reference angle may be used as the loss function. On the other hand, in the case illustrated in FIG. 9B, it is advantageous to use the position determination mask Mp. FIG. 9B is a diagram showing an example in which the use of a mask is advantageous for calculating the loss function. The part P included in the correction patch image Ipc shown in FIG. 9B has a zigzag shape, and it is difficult to appropriately obtain the principal axis angle from the moment of the image of this part P. Therefore, from the viewpoint of handling parts P of various shapes, the position determination mask Mp is used here.

[0067] In step S606, a patch image Ip (test data) that is reserved for testing and not used for learning among the patch images Ip stored in the patch image list is forward propagated to the alignment neural network with updated parameters to calculate a correction amount. Then, based on this correction amount, a loss function is calculated using the position determination mask Mp corresponding to the test data in the same manner as in steps S603 to S604 above.

[0068] The calculation unit 31 stores the loss function calculated in step S606 each time step S606 is executed, and calculates the minimum value of the multiple loss functions thus stored. Then, the calculation unit 31 checks whether the most recently calculated loss function has updated the minimum value. In particular, in step S607, it is determined whether the minimum value has not been updated for 10 consecutive times, that is, whether a loss function larger than the minimum value has been calculated. Then, if a loss function equal to or smaller than the minimum value has been calculated in the past 10 times (if "NO" in step S607), the process returns to step S601, whereas if a loss function larger than the minimum value has been calculated for 10 consecutive times (if "YES" in step S607), the flowchart in FIG. 9A ends. Note that the number of times is not limited to 10 times, and can be changed as appropriate as necessary.

[0069] In the above-described grasping inference, when the corrected patch image Ipc is input to the grasping classification network unit 47, the grasping classification network unit 47 calculates a grasping success probability when the part P included in the corrected patch image Ipc is grasped by the robot hand 51 at the position indicated by the corrected patch image Ipc. In particular, the grasping classification network unit 47 calculates the grasping success probability from the corrected patch image Ipc using a grasping classification neural network. Next, a method of making the grasping classification neural network learn the relationship between the corrected patch image Ipc and the grasping success probability will be described.

[0070] 10A to 10C are an example of a flowchart for causing the grip classification neural network to execute learning. This flowchart is executed by the calculation unit 31 of the control device 3. As in the case of learning the above-mentioned alignment neural network, a simulator for constructing the virtual part gripping system 1 is used for learning the grip classification neural network.

[0071] In the flowchart of Fig. 10A, learning data is collected in the same manner as in Fig. 8A. That is, steps S701 to S709 in Fig. 10A are the same as steps S501 to S509 in Fig. 8A, except for the following point. That is, in step S701, it is determined whether the number of learning operations has reached a predetermined number, not the number of data acquisitions. This predetermined number can be set in advance by an operator, for example.

[0072] 10A, when one patch image Ip having an appropriate object area is selected by executing steps S701 to S709, the alignment network unit 45 calculates a correction amount corresponding to the patch image Ip using the alignment neural network that has completed the above learning (step S710), and stores the patch image Ip and the correction amount in a correction amount list in association with each other (step S711). Then, steps S708 to S711 are repeated until the count value reaches a maximum (until "YES" is obtained in step S712), and pairs of the patch image Ip and the correction amount are sequentially stored in the correction amount list. When the count value reaches a maximum ("YES" in step S712), the process proceeds to step S712 in FIG. 10B.

[0073] In step S712, the alignment network unit 45 executes a process of correcting the cut-out range Rc of the patch image Ip based on the correction amount to generate a corrected cut-out range Rcc, and generating a corrected patch image Ipc based on the corrected cut-out range Rcc for each pair of the patch image Ip and the correction amount stored in the correction amount list. In this way, a plurality of correction patch images Ipc are generated. The specific procedure for generating the correction patch image Ipc is as described above.

[0074] In step S713, it is confirmed whether the number of data necessary for learning has been acquired. This necessary number can be preset by, for example, an operator. If the necessary number of data has been acquired (if "YES" in step S713), the process proceeds to step S717 (FIG. 10C) described later, whereas if the number of acquired data is less than the necessary number (if "NO" in step S713), the process proceeds to step S714.

[0075] In step S714, one correction patch image Ipc is randomly selected (for example, based on the output of a random number generator) from among the multiple correction patch images Ipc generated in step S712. Then, in step S715, in the virtual part gripping system 1, gripping of the part P included in the one correction patch image Ipc is attempted by the robot hand 51 located at the position of the one correction patch image Ipc. Note that the position of the correction patch image Ipc corresponds to the position of the correction cut-out range Rcc into which the correction patch image Ipc is cut out. Then, the result of the gripping attempt (1 if successful, 0 if unsuccessful) is stored in a success / failure result list in association with the one correction patch image Ipc (step S716), and the process returns to step S701 in FIG. 10A.

[0076] On the other hand, if it is determined in step S713 that the necessary amount of data has been acquired (YES), the process proceeds to step S717 in Fig. 10C as described above. In step S717, a horizontally inverted correction patch image Ipc obtained by inverting the correction patch image Ipc horizontally, a vertically inverted correction patch image Ipc obtained by inverting the correction patch image Ipc vertically, and a vertically inverted correction patch image Ipc obtained by inverting the correction patch image Ipc horizontally and vertically are generated. As a result, three types of images, the horizontally inverted correction patch image Ipc, the vertically inverted correction patch image Ipc, and the vertically inverted correction patch image Ipc, are prepared for each correction patch image Ipc in the success / failure result list. In other words, a number of correction patch images Ipc that is three times the number of correction patch images Ipc stored in the success / failure result list are prepared.

[0077] In step S718, each of the multiple correction patch images Ipc generated in step S717 is forward propagated in the grip classification neural network of the grip classification network unit 47, and a grip success probability is calculated for each correction patch image Ipc. Then, in step S719, an average value of the grip success probability for each of the horizontally inverted correction patch image Ipc, the vertically inverted correction patch image Ipc, and the vertically and horizontally inverted correction patch image Ipc generated from the same correction patch image Ipc is calculated. As a result, an average value of the grip success probability is calculated for each correction patch image Ipc stored in the success / failure result list.

[0078] In step S720, a random number generator generates one of the values ​​"0", "1" and "2". If the random number generation results in "0", one correction patch image Ipc is randomly selected from among the correction patch images Ipc whose grasping success probability was calculated in step S719 (step S721), if the random number generation results in "1", one correction patch image Ipc whose grasping success probability is closest to "0.5" (in other words, 50%) is selected from among the correction patch images Ipc (step S722), and if the random number generation results in "2", one correction patch image Ipc whose grasping success probability is highest is selected from among the correction patch images Ipc (step S723).

[0079] In step S724, in the virtual part gripping system 1, gripping of the part P indicated by one of the correction patch images Ipc is attempted by the robot hand 51 located at the position of the one of the correction patch images Ipc. Then, a loss function is calculated based on the result of part gripping success or failure (1 for success, 0 for failure) and the average gripping success probability calculated for the one of the correction patch images Ipc in step S719. As the loss function, various well-known functions such as cross entropy error can be used.

[0080] Moreover, the calculation unit 31 stores the loss function calculated in step S725 every time step S725 is executed, and calculates the minimum value of the plurality of loss functions thus stored. Then, the calculation unit 31 checks whether the most recently calculated loss function has updated the minimum value. In particular, in step S726, it is determined whether the minimum value has not been updated for 10 consecutive times, that is, whether a loss function larger than the minimum value has been calculated. Then, if a loss function equal to or smaller than the minimum value has been calculated in the past 10 times (if "NO" in step S726), the grasping success / failure result in step S724 is stored in a success / failure result list in association with one correction patch image Ipc (step S727). Then, in step S728, the parameters of the grasping classification neural network are updated by backpropagating the loss function calculated in step S725 in the grasping classification neural network (error backpropagation). On the other hand, if a loss function greater than the minimum value is calculated ten consecutive times in step S726 (if "NO"), the process returns to step S701 in Fig. 10A. Note that the number of times is not limited to ten, and can be changed as necessary.

[0081] In the embodiment described above, a composite image Ic is generated (step S201) by combining grayscale images Ig (brightness images) and depth images Id each showing a plurality of parts P. In the composite image Ic generated in this manner, the shapes of parts P at relatively high positions among the plurality of parts P tend to remain, and this is useful for recognizing such parts P.

[0082] Furthermore, based on the result of recognizing the part P in this way, the robot hand 51 grasps the part P (step S104). Therefore, it becomes possible for the robot hand 51 to grasp the part with a high success rate.

[0083] Moreover, the grayscale image Ig indicates the brightness Vg for each pixel PX, the depth image Id indicates the depth Vd for each pixel PX, and the composite image Ic indicates the composite value Vc for each pixel PX. In particular, the image composition unit 41 generates the composite image Ic by calculating the composite value Vc in which the brightness Vg is weighted by the depth Vd. The composite image Ic generated in this manner is likely to retain the shape of a part P at a relatively high position among multiple parts P, and is useful for recognizing such parts P.

[0084] Moreover, the image synthesis unit 41 uses the following formula Vc = Vd × (1 + Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the grayscale image Ig The composite image Ic thus generated is useful for recognizing a part P that is located at a relatively high position among a plurality of parts P, since the composite image Ic is likely to retain the shape of the part P that is located at a relatively high position among a plurality of parts P.

[0085] Furthermore, a corrected cutout range Rcc (hand range) indicating the range of the robot hand 51 is set for the part P shown in the composite image IC, and a corrected patch image Ipc (target part image) indicating the part P included in the corrected cutout range Rcc is cut out from the composite image Ic, thereby generating a plurality of corrected patch images Ipc each indicating a different part P (steps S301 to S306). Furthermore, a gripping success probability when the robot hand 51 attempts to grip the part P in the positional relationship between the robot hand 51 and the part P indicated by the corrected patch image Ipc is calculated for each of the plurality of corrected patch images Ipc (step S307). As described above, the composite image Ic used at this time is likely to retain the shape of the part P at a relatively high position. Therefore, it is possible to reliably generate a corrected patch image Ipc including a part P at a relatively high position, in other words, a part P with a relatively high gripping success probability.

[0086] Furthermore, one correction patch image Ipc is selected from the multiple correction patch images Ipc based on the grasping success probability calculated for each of the multiple correction patch images Ipc, and it is determined that the part P indicated by the selected correction patch image Ipc is to be grasped by the robot hand 51 in the posture indicated by the selected correction patch image Ipc (steps S401 to S406). With this configuration, it is possible to reduce the frequency of failure in grasping the part P by the robot hand 51.

[0087] Furthermore, among the multiple correction patch images Ipc that do not satisfy a predetermined exclusion condition (step S405), the correction patch image Ipc with the highest probability of successful grasping is selected as one correction patch image Ipc (steps S401 to S406). With this configuration, it is possible to reliably reduce the frequency of failure to grasp the part P by the robot hand 51.

[0088] Specifically, the predetermined exclusion condition is that the distance between the edge of the compartment storage 911 (container) of the part bin 91 and the position of the correction patch image Ipc is less than a predetermined value (step S406). This makes it possible to prevent the occurrence of a situation in which gripping of the part P fails due to the influence of the wall surface at the edge of the compartment storage 911.

[0089] The grip classification network unit 47 also has a grip classification neural network that learns the relationship between the corrected patch image Ipc and the grip success probability using as training data the success or failure of gripping when the robot hand 51 attempts to grip the part P in the positional relationship between the robot hand 51 and the part P indicated by the corrected patch image Ipc. The grip classification neural network then calculates the grip success probability for each of the multiple corrected patch images Ipc. With this configuration, it is possible to reliably reduce the frequency of failure to grip the part P by the robot hand 51 by using the learning results of the grip classification neural network.

[0090] Furthermore, this embodiment can be said to be suitable for learning by such a grip classification neural network. That is, as an example of a learning method of the grip classification neural network, it is possible to input both the grayscale image Ig and the depth image Id to the grip classification neural network and have the grip classification neural network learn the relationship between these images Ig, Id and the grip success probability. However, it is difficult for the grip classification neural network to learn the grip success probability of the robot hand 51 from the association between the brightness of the part P shown in the grayscale image Ig and the height of the part P shown in the depth image Id, and the calculation load is also large. On the other hand, in this embodiment, a composite image Ic (corrected patch image Ipc) obtained by combining the grayscale image Ig and the depth image Id is input to the grip classification neural network, so that the grip classification neural network can learn based on this composite image Ic. As a result, learning can be performed relatively easily and the calculation load can be reduced.

[0091] Furthermore, gripping state information (a side view image and mass of the part P) indicating a state in which the part P indicated by one of the correction patch images Ipc is actually gripped by the robot hand 51 in the posture indicated by the one of the correction patch images Ipc is acquired (steps S105, S112). Then, based on this gripping state information, it is determined whether the robot hand 51 has successfully gripped the part P (steps S107, S109, S113). With this configuration, it is possible to confirm whether the gripping of the selected part P has actually been successful based on the gripping success probability calculated for the correction patch image Ipc.

[0092] In this manner, in the above embodiment, the part gripping system 1 corresponds to an example of the "part gripping system" of the present invention, the control device 3 corresponds to an example of the "image processing device" of the present invention, the main control unit 311 corresponds to an example of each of the "image acquisition unit" and "success / failure determination unit" of the present invention, the image synthesis unit 41 corresponds to an example of the "image synthesis unit" of the present invention, the patch image generation unit 43, the alignment network unit 45 and the gripping classification network unit 47 cooperate to function as an example of the "success probability calculation unit" of the present invention, the robot hand 51 corresponds to an example of the "robot hand" of the present invention, the camera 83 corresponds to an example of the "camera" of the present invention, the compartment storage 911 of the part bin 91 corresponds to an example of a "container" of the present invention, the depth image Id corresponds to an example of the "depth image" of the present invention, the grayscale image Ig corresponds to an example of the "brightness image" of the present invention, the composite image Ic corresponds to an example of the "composite image" of the present invention, the corrected patch image Ipc corresponds to an example of the "target part image" of the present invention, and the corrected cut-out range Rcc corresponds to an example of the "hand range" of the present invention.

[0093] The present invention is not limited to the above embodiment, and various modifications can be made to the above without departing from the spirit of the present invention. For example, in step S105, the part P held by the robot hand 51 may be imaged by the camera 83 from different directions to obtain multiple side view images. These side view images can be obtained, for example, by imaging the part P while rotating the robot hand 51 holding the part P in the θ direction. This allows the number of parts P to be confirmed in step S107 and abnormalities in the part P (too small an area) to be confirmed in step S109 from multiple directions.

[0094] Moreover, the learning of the grip classification neural network may be performed by executing the flowchart of Fig. 11. Here, Fig. 11 is a flowchart showing an example of a method for re-learning the grip classification neural network of the grip classification network unit. This flowchart is executed by the main control unit 311, for example, when the planned bin picking is completed.

[0095] In step S801, the main control unit 311 checks the history of detection of anomalies based on side view images (steps S107 and S108: NO) or mass measurement (step S113: NO) in bin picking performed in the past. If the number of times anomalies have been detected is equal to or greater than a predetermined number (step S802: YES), re-learning of the grip classification neural network of the grip classification network unit 47 is executed (step S803). In this re-learning, the corrected patch image Ipc indicating the part P in which an abnormality has been detected and the grip success / failure result (i.e., failure) are used as teacher data. Specifically, an error function is calculated based on the grip success / failure probability and the grip success / failure result (failure) obtained by forward propagating the corrected patch image Ipc through the grip classification neural network, and the error function is back-propagated through the grip classification neural network to update the parameters of the grip classification neural network (re-learning).

[0096] 11, the grip classification neural network is made to re-learn based on the results of acquiring grip state information (side-view image, mass) for the part P gripped by the robot hand 51. In this configuration, the grip classification neural network is re-learned in accordance with the actual success or failure of gripping the part P selected based on the grip success probability calculated for the corrected patch image Ipc, and the accuracy of the calculation of the grip success probability by the grip classification neural network can be improved.

[0097] 12 is a modified example of the grasp classification neural network of the grasp classification network section. In this grasp classification neural network 471, a multi-layer convolutional neural network 472 and a fully connected layer 473 are arranged in series. In addition, a spatial attention module 474 and a channel attention module 475 are provided on the output side of each convolutional neural network 472, and the feature map output from the convolutional neural network 472 is input to the next convolutional neural network 472 or the fully connected layer 473 after being weighted by the spatial attention module 474 and the channel attention module 475.

[0098] In particular, the attention mask Ma that the spatial attention module 474 adds to the feature map has two attention regions Pg and Pp that pass through the center position of the corrected patch image Ipc (in other words, the corrected cutout range Rcc). That is, in the attention mask Ma, the weights of the attention region Pg and the attention region Pp are larger than the weights of the other regions, and these weights are added to the feature map. Here, the attention region Pg is parallel to the gripping direction G, and the attention region Pp is orthogonal to the gripping direction G. In particular, as in the above example, when the long axis direction of the part P is orthogonal to the gripping direction G, the attention region Pp is parallel to the long axis direction of the part P. That is, this attention mask Ma focuses on the attention region Pp corresponding to the ideal position of the part P in the corrected patch image Ipc and the attention region Pg corresponding to the path along which the claws 511 of the robot hand 51 approach the part P.

[0099] In the grip classification neural network, the attention mask Ma configured as above is added to the feature map output from the convolutional neural network 472, and the feature map is weighted. Therefore, the angle of the long axis direction of the part P with respect to the grip direction G and the state of the movement path of the robot hand 51 gripping the part P (presence or absence of other parts) can be accurately reflected in the judgment by the grip classification neural network.

[0100] Furthermore, the method of generating the composite image Ic is not limited to the example using the above formula, and the composite image Ic may be generated using another formula that calculates the composite value Vc of the composite image Ic by weighting the brightness Vg of the grayscale image Ig by the depth Vd of the depth image Id.

[0101] In the above example, the grayscale image Ig and the depth image Id are combined to generate the combined image Ic. In this case, the combined image Ic may be generated by combining an inverted grayscale image Ig (luminance image) obtained by inverting the luminance of the grayscale image Ig with the depth image Id. In particular, when gripping a part P having a surface that has been subjected to black plating, it is preferable to generate the combined image Ic using the inverted grayscale image Ig.

[0102] In addition, it is not necessary to extract the patch image Ip from the binarized composite image Ic, and the patch image Ip may be extracted from the composite image Ic without performing binarization. The same applies to the correction patch image Ipc.

[0103] Furthermore, various modes for setting the cut-out range Rc for the part P in patch image processing can be assumed. For example, the cut-out range Rc can be set so that the geometric center of gravity of the cut-out range Rc coincides with the geometric center of gravity of the part P. However, this is not limiting, and in short, it is sufficient that the cut-out range Rc is set so as to include the target part P.

[0104] Furthermore, the specific configuration of the robot hand 51 is not limited to the above example. For example, the number of claws 511 of the robot hand 51 is not limited to two, and may be three or more. Also, a robot hand 51 that attracts by negative pressure or magnetic force may be used. Even in these cases, it is possible to set the cut-out range Rc to the range to be grasped by the robot hand 51, and cut out the patch image Ip from the cut-out range Rc.

[0105] In the above embodiment, the correction amount (Δx, Δy, Δθ) of the cut-out range Rc into which the patch image Ip is cut out is calculated by the positioning network unit 45, and the corrected patch image Ipc cut out in the corrected cut-out range Rcc obtained by correcting the cut-out range Rc by the correction amount (Δx, Δy, Δθ) is input to the grasping classification network unit 47. However, the patch image Ip may be directly input to the grasping classification network unit 47 without being corrected by the correction amount (Δx, Δy, Δθ), and the grasping classification network unit 47 may be configured to calculate the grasping success probability directly from the patch image Ip. [Explanation of symbols]

[0106] 1...Part gripping system 3...Control device (image processing device) 311...Main control unit (image acquisition unit, success / failure determination unit) 41...Image synthesis unit 43...Patch image generation unit (success probability calculation unit) 45...Alignment network unit (success probability calculation unit) 47... Grasping classification network unit (success probability calculation unit) 51...Robot hand 83...Camera 91…Parts bin 911… Compartment storage (container) Id…depth image Ig...Grayscale image (brightness image) Ic…Composite image Ipc...Correction patch image (target part image) Rcc…Correction range (hand range)

Claims

1. an image acquisition unit that acquires a luminance image showing a plurality of parts stored in a container and a depth image showing the plurality of parts; an image synthesis unit that generates a synthetic image by synthesizing the luminance image and the depth image acquired by the image acquisition unit; Equipped with the luminance image indicates a luminance Vg for each pixel; the depth image indicates a depth Vd for each pixel; The composite image indicates a composite value Vc for each pixel, The image synthesis unit is Vc=Vd×(1+Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the luminance image The image processing device generates the composite image by calculating a composite value Vc in which the luminance Vg is weighted by the depth Vd based on the above.

2. 2. The image processing device according to claim 1, further comprising a success probability calculation unit that generates a plurality of target part images each indicating a different part by setting a hand range indicating a range of a robot hand for the part and creating a target part image indicating the part included in the hand range based on the composite image, and calculates a gripping success probability for each of the plurality of target part images when the robot hand attempts to grip the part in the positional relationship between the robot hand and the part indicated by the target part image.

3. 3. The image processing device according to claim 2, wherein the success probability calculation unit selects one target part image from the plurality of target part images based on the grasping success probability calculated for each of the plurality of target part images, and determines that the part indicated by the one target part image is to be grasped by the robot hand in the posture indicated by the one target part image.

4. The image processing device according to claim 3 , wherein the success probability calculation unit selects, as the one target part image, the target part image having the highest grasping success probability from among the plurality of target part images that do not satisfy a predetermined exclusion condition.

5. The image processing apparatus according to claim 4 , wherein the predetermined exclusion condition is that a distance between an edge of the container and the target part image is less than a predetermined value.

6. 6. The image processing device according to claim 3, wherein the success probability calculation unit has a neural network that has learned the relationship between the target part image and the grasping success probability using as training data the success or failure of grasping when the robot hand attempts to grasp the part in the positional relationship between the robot hand and the part shown in the target part image, and calculates the grasping success probability for each of the multiple target part images using the neural network.

7. The image processing device according to claim 6, further comprising a success / failure determination unit that acquires gripping state information indicating a state in which the part indicated by the one target part image is gripped by the robot hand in the posture indicated by the one target part image, and determines whether the part has been successfully gripped by the robot hand based on the gripping state information.

8. The image processing device according to claim 7 , wherein the success / failure determination unit acquires, as the gripping state information, a mass measured by a mass meter when the part gripped by the robot hand from the container is placed on the mass meter.

9. The image processing device according to claim 7 , wherein the success / failure determination unit acquires, as the gripping state information, an image of the part being gripped by the robot hand captured by a camera.

10. The image processing device according to claim 9 , wherein the success / failure determination unit acquires, as the gripping state information, a plurality of images of the part gripped by the robot hand captured by the camera from different directions.

11. 11. The image processing device according to claim 7, wherein the success probability calculation unit causes the neural network to perform re-learning based on a result of acquiring the gripping state information for the part gripped by the robot hand.

12. An image processing device according to any one of claims 3 to 11, a robot hand that selects one target part image from the plurality of target part images, and grasps the part indicated by the one target part image with the robot hand in a posture indicated by the one target part image, based on a decision of the image processing device; A part gripping system comprising:

13. acquiring a brightness image showing a plurality of parts contained in a container and a depth image showing the plurality of parts; generating a composite image by combining the luminance image and the depth image; Equipped with the luminance image indicates a luminance Vg for each pixel; the depth image indicates a depth Vd for each pixel; The composite image indicates a composite value Vc for each pixel, In the step of generating the composite image, Vc=Vd×(1+Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the luminance image The image processing method generates the composite image by calculating a composite value Vc in which the luminance Vg is weighted by the depth Vd based on the above.

14. acquiring a brightness image showing a plurality of parts contained in a container and a depth image showing the plurality of parts; generating a composite image by combining the luminance image and the depth image; a step of generating a plurality of target part images each showing a different part by setting a hand range indicating a range of a robot hand for the part and creating a target part image indicating the part included in the hand range based on the composite image, and calculating a grasping success probability for each of the plurality of target part images when the robot hand attempts to grasp the part in a positional relationship between the robot hand and the part indicated by the target part image; selecting one target part image from among the plurality of target part images based on the grasping success probability calculated for each of the plurality of target part images, and determining that the part indicated by the one target part image is to be grasped by the robot hand in a posture indicated by the one target part image; selecting one target part image from the plurality of target part images, and grasping the part by the robot hand based on a decision that the part indicated by the one target part image is to be grasped by the robot hand in a posture indicated by the one target part image; Equipped with the luminance image indicates a luminance Vg for each pixel; the depth image indicates a depth Vd for each pixel; The composite image indicates a composite value Vc for each pixel, In the step of generating the composite image, Vc=Vd×(1+Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the luminance image The part gripping method generates the composite image by calculating a composite value Vc in which the brightness Vg is weighted by the depth Vd based on the above.

Images