Image processing device, part gripping system, image processing method, and part gripping method

Abstract

To provide a technique useful for recognizing parts located at relatively high positions among a plurality of parts stored in a container.SOLUTION: A grayscale image Ig (brightness image) and a depth image Id, each showing multiple parts P, are combined to generate a composite image Ic (step S201). In the composite image Ic thus generated, shapes of parts P at relatively high positions, among the multiple parts P, tend to remain, which is useful for recognizing such parts P.SELECTED DRAWING: Figure 4B

Description

[Technical Field]

[0001] The present invention relates to a technique for gripping a plurality of parts stored in a container with a robot hand, and is particularly suitable for use in bin picking. [Background technology]

[0002] In Patent Document 1, when a robot hand grasps randomly stacked parts, an image of the parts is used to recognize the parts to be grasped. Specifically, an area having a brightness equal to or greater than a predetermined threshold is identified from the image of the piled parts, and point cloud data is generated for this area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-147256 Summary of the Invention [Problem to be solved by the invention]

[0004] In bin picking, to reliably grasp parts, it is preferable to target the parts closest to the robot hand, i.e., the parts at the highest position, among the multiple piled parts. However, the method of Patent Document 1 generates point cloud data by limiting the area of ​​the captured image that has a brightness equal to or greater than a threshold. Therefore, if the brightness of the captured image does not accurately reflect the height of the parts, the generated point cloud data is not necessarily useful for recognizing parts at high positions.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique useful for recognizing a part located at a relatively high position among a plurality of parts stored in a container. [Means for solving the problem]

[0006] The image processing device of the present invention includes an image acquisition unit that acquires a brightness image showing multiple parts stored in a container and a depth image showing the multiple parts, and an image synthesis unit that generates a synthesized image by synthesizing the brightness image and depth image acquired by the image acquisition unit.

[0007] The image processing method of the present invention includes a step of acquiring a brightness image showing multiple parts stored in a container and a depth image showing the multiple parts, and a step of generating a composite image by combining the brightness image and the depth image.

[0008] In the image processing device and method configured as described above, a composite image is generated by combining brightness images and depth images that respectively represent multiple components. In the composite image generated in this manner, the shapes of components that are relatively high up among the multiple components tend to remain, which is useful for recognizing such components.

[0009] The image processing device may also be configured so that the luminance image indicates the luminance Vg for each pixel, the depth image indicates the depth Vd for each pixel, and the composite image indicates the composite value Vc for each pixel, and the image synthesis unit generates the composite image by calculating the composite value Vc by weighting the luminance Vg by the depth Vd. The composite image generated in this manner tends to retain the shapes of components at relatively high positions among multiple components, and is useful for recognizing such components.

[0010] The image synthesis unit is Vc = Vd × (1 + Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the luminance image The image processing device may be configured to generate a composite image based on the above. The composite image generated in this manner is likely to retain the shapes of components that are located at relatively high positions among multiple components, and is useful for recognizing such components.

[0011] The image processing device may further include a success probability calculation unit that sets a hand range indicating the range of the robot hand for each part and creates target part images indicating parts included in the hand range based on the composite image to generate multiple target part images each indicating a different part, and calculates the probability of successful grasping for each of the multiple target part images when the robot hand attempts to grasp the part based on the positional relationship between the robot hand and the part indicated by the target part image. In this configuration, a hand range indicating the range of the robot hand is set for the part included in the composite image, target part images indicating parts included in the hand range are created based on the composite image, and the probability of successful grasping for each target part image is calculated. As described above, the composite image used in this process tends to retain the shapes of parts located at relatively high positions. Therefore, it is possible to reliably generate target part images including parts located at relatively high positions, in other words, parts with a relatively high probability of successful grasping.

[0012] The image processing device may be configured so that the success probability calculation unit selects one target part image from the plurality of target part images based on the grasp success probability calculated for each of the plurality of target part images, and determines that the part indicated by the one target part image is to be grasped by the robot hand in the posture indicated by the one target part image. With such a configuration, it is possible to reduce the frequency of failure in grasping a part by the robot hand.

[0013] The image processing device may be configured such that the success probability calculation unit selects, from among a plurality of target part images that do not satisfy a predetermined exclusion condition, the target part image with the highest probability of successful grasping as the single target part image. With this configuration, it is possible to reliably reduce the frequency of failures in grasping a part by the robot hand.

[0014] For example, the predetermined exclusion condition is that the distance between the edge of the container and the target part image is less than a predetermined value, which can prevent the occurrence of a situation where the gripping of the part fails due to the influence of the wall surface of the edge of the container.

[0015] The image processing device may also be configured so that the success probability calculation unit has a neural network that has learned the relationship between the target part image and the probability of grasping success using training data on the success or failure of grasping when the robot hand attempts to grasp a part in the positional relationship between the robot hand and the part shown in the target part image, and calculates the probability of grasping success for each of the multiple target part images using the neural network. With this configuration, it is possible to reliably reduce the frequency of failure in grasping parts by the robot hand by using the learning results of the neural network.

[0016] Furthermore, the present invention can be said to be suitable for such neural network learning. In other words, one example of a neural network learning method is to input both a brightness image and a depth image to the neural network and have the neural network learn the relationship between these images and the grasping success probability. However, learning the grasping success probability of a robot hand from the association between the brightness of a part indicated by the brightness image and the height of the part indicated by the depth image is difficult for the neural network and requires a large computational load. On the other hand, in the present invention, a composite image obtained by combining the brightness image and the depth image is input to the neural network, so the neural network can learn based on this composite image. As a result, learning can be performed relatively easily and the computational load can be reduced.

[0017] The image processing device may be configured to further include a success / failure determination unit that acquires gripping state information indicating a state in which a part indicated by a target part image is grasped by a robot hand in a posture indicated by the target part image, and determines whether the robot hand has successfully grasped the part based on the gripping state information. With this configuration, it is possible to confirm whether the selected part has been successfully grasped based on the grasp success probability calculated for the target part image.

[0018] Note that various specific contents of the gripping state information can be assumed. For example, the success / failure determination unit can acquire, as the gripping state information, the mass measured by a mass meter when the robotic hand places the part gripped from the container on the mass meter. Alternatively, the success / failure determination unit can acquire, as the gripping state information, an image of the part gripped by the robotic hand captured by a camera. In the latter case, the success / failure determination unit may acquire, as the gripping state information, multiple images of the part gripped by the robotic hand captured by a camera from different directions.

[0019] The image processing device may also be configured so that the success probability calculation unit causes the neural network to re-learn based on the results of acquiring grip state information for a part to be gripped by the robot hand. In this configuration, the neural network is re-learned in response to the results of the success or failure of gripping of the part selected based on the grip success probability calculated for the target part image, thereby improving the accuracy of the calculation of the grip success probability by the neural network.

[0020] The part gripping system of the present invention comprises the above-mentioned image processing device, and a robot hand that grips a part based on the image processing device's decision to select one target part image from a plurality of target part images and grip the part indicated by the one target part image with the robot hand in the posture indicated by the one target part image.

[0021] The part gripping method of the present invention includes the steps of: acquiring a brightness image showing a plurality of parts stored in a container and a depth image showing the plurality of parts; generating a composite image by combining the brightness image and the depth image; generating a plurality of target part images each showing a different part by setting a hand range indicating the range of the robot hand for the part and creating target part images showing the parts included in the hand range based on the composite image; calculating, for each of the plurality of target part images, the probability of successful gripping when the robot hand attempts to grip the part in the positional relationship between the robot hand and the part indicated by the target part image; selecting one target part image from the plurality of target part images based on the probability of successful gripping calculated for each of the plurality of target part images, and determining that the part indicated by the one target part image will be gripped by the robot hand in the pose indicated by the one target part image; and selecting one target part image from the plurality of target part images and gripping the part with the robot hand based on the determination that the part indicated by the one target part image will be gripped by the robot hand in the pose indicated by the one target part image.

[0022] In the part grasping system and method configured in this manner, a composite image is generated by combining brightness images and depth images each representing a plurality of parts. The composite image thus generated tends to retain the shapes of parts located at relatively high positions among the plurality of parts, making it useful for recognizing such parts. Furthermore, the robotic hand grasps the parts based on the results of this part recognition. Therefore, it is possible to grasp parts with a high success rate using the robotic hand. [Effects of the Invention]

[0023] According to the present invention, it is possible to obtain an image (composite image) that is useful for recognizing a component that is located at a relatively high position among a plurality of components stored in a container. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a plan view schematically showing an example of a part gripping system according to the present invention. [Figure 2]FIG. 2 is a perspective view schematically showing a robot hand used to grip a part in the part gripping system of FIG. 1. [Figure 3] FIG. 2 is a block diagram showing an example of an electrical configuration of a control device. [Figure 4A] 4 is a flowchart showing an example of bin picking executed by the part gripping system of FIG. 1. [Figure 4B] 4B is a flowchart showing an example of patch image processing performed in the bin picking of FIG. 4A. [Figure 4C] 4B is a flowchart showing an example of grasping inference performed in the bin picking of FIG. 4A. [Figure 4D] 4D is a flowchart showing an example of determination of a part to be grasped, which is executed in the grasp inference of FIG. 4C. [Figure 5A] FIG. 4C is a diagram schematically illustrating operations performed in the patch image processing of FIG. 4B. [Figure 5B] FIG. 4C is a diagram schematically illustrating operations performed in the patch image processing of FIG. 4B. [Figure 5C] FIG. 4C is a diagram schematically illustrating operations performed in the patch image processing of FIG. 4B. [Figure 5D] FIG. 4C is a diagram schematically illustrating operations performed in the patch image processing of FIG. 4B. [Figure 5E] FIG. 4C is a diagram schematically illustrating operations performed in the patch image processing of FIG. 4B. [Figure 6A] FIG. 4D is a diagram schematically showing the operation executed in the grasp inference of FIG. 4C. [Figure 6B] FIG. 4D is a diagram schematically showing the operation executed in the grasp inference of FIG. 4C. [Figure 6C] FIG. 4D is a diagram schematically showing the operation executed in the grasp inference of FIG. 4C. [Figure 7] FIG. 4D is a diagram schematically showing the operation executed in the grasp inference of FIG. 4C. [Figure 8A] 10 is a flowchart showing an example of a method for collecting training data for an alignment neural network. [Figure 8B] FIG. 10 is a diagram schematically showing an example of a position determination mask generated from a patch image. [Figure 9A]8B is an example of a flowchart for training an alignment neural network on the training data collected in FIG. 8A. [Figure 9B] FIG. 10 is a diagram schematically illustrating an example in which the use of a mask is advantageous for calculating a loss function. [Figure 10A] 1 is an example of a flowchart for training a grasp classification neural network. [Figure 10B] 1 is an example of a flowchart for training a grasp classification neural network. [Figure 10C] 1 is an example of a flowchart for training a grasp classification neural network. [Figure 11] 10 is a flowchart showing an example of a method for relearning the grasp classification neural network of the grasp classification network unit. [Figure 12] 10 is a modified example of the grasp classification neural network of the grasp classification network section. DETAILED DESCRIPTION OF THE INVENTION

[0025] FIG. 1 is a plan view that schematically shows an example of a part gripping system according to the present invention, and FIG. 2 is a perspective view that schematically shows a robot hand used to grip parts in the part gripping system of FIG. 1. In these figures and the following figures, the horizontal X direction, the horizontal Y direction that is perpendicular to the X direction, and the vertical Z direction are indicated as appropriate. These X, Y, and Z directions form a global coordinate system. As shown in FIG. 1, the part gripping system 1 includes a control device 3 and a work robot 5, and the work robot 5 performs a task (bin picking) under the control of the control device 3.

[0026] Specifically, a parts bin 91 and a kitting tray 92 are placed in the workspace of the work robot 5. The parts bin 91 has a plurality of storage compartments 911 for storing parts, and a large number of parts are piled up in each storage compartment 911. The kitting tray 92 has a plurality of storage compartments 921 for storing parts, and a predetermined number of parts are placed in each storage compartment 921. The work robot 5 then picks up a part from the storage compartment 911 of the parts bin 91 (bin picking) and transfers it to the storage compartment 921 of the kitting tray 92. In addition, a trash can 93 is placed between the parts bin 91 and the kitting tray 92, and if a defective part is detected, the work robot 5 discards the defective part in the trash can 93.

[0027] The work robot 5 is a scalar robot equipped with a robot hand 51 attached to its tip. By grasping a part with the robot hand 51 and moving it, the robot hand 51 transfers the part from a parts bin 91 to a kitting tray 92 or disposes of the part in a trash can 93. As shown in FIG. 2 , the robot hand 51 has degrees of freedom in the X, Y, Z, and θ directions. Here, the θ direction is the direction of rotation about a rotation axis parallel to the Z direction. The robot hand 51 also has two claws 511 arranged in a gripping direction G, each of which has a flat plate shape perpendicular to the gripping direction G. The robot hand 51 can expand or contract the spacing between the two claws 511 in the gripping direction G, and grasps the part by pinching it with these claws 511 in the gripping direction G. Note that in FIG. 2 , the gripping direction G is parallel to the X direction; however, depending on the position of the robot hand 51 in the θ direction, the gripping direction G may be tilted relative to the X direction.

[0028] The part gripping system 1 further includes two cameras 81 and 83 and a mass meter 85. The camera 81 is a plan-view camera that captures images of the multiple parts stacked in the compartment storage 911 of the part bin 91 from the Z direction (above), and faces the workspace of the work robot 5 from the Z direction. The camera 81 captures a grayscale image (two-dimensional image) that indicates the brightness of the imaged object (part) and a depth image (three-dimensional image) that indicates the distance to the imaged object. Specific methods for acquiring the depth image include a phase shift method and a stereo matching method. The camera 83 is a side-view camera that captures images of the part gripped by the robot hand 51 from the Y direction and is attached horizontally to the base of the robot hand 51. The camera 83 captures a grayscale image (two-dimensional image) that indicates the brightness of the imaged object (part). The mass meter 85 measures the mass of the part placed in the compartment storage 921 of the kitting tray 92.

[0029] FIG. 3 is a block diagram showing an example of the electrical configuration of the control device. The control device 3 is, for example, a personal computer equipped with a calculation unit 31, a storage unit 35, and a UI (User Interface) 39. The calculation unit 31 is, for example, a processor equipped with a CPU (Central Processing Unit) or the like, and includes a main control unit 311 and an image processing unit 4. The main control unit 311 and image processing unit 4 are deployed in the calculation unit 31 by executing a predetermined program. The main control unit 311 controls hardware such as the robot hand 51, cameras 81 and 83, and mass meter 85, and the image processing unit 4 performs image processing to recognize the part to be grasped by the robot hand 51. In particular, the image processing unit 4 includes an image synthesis unit 41, a patch image generation unit 43, an alignment network unit 45, and a grasp classification network unit 47. These functions will be described in detail below.

[0030] The storage unit 35 is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores, for example, programs and data for deploying the main control unit 311 or the image processing unit 4 in the calculation unit 31. The UI 39 has input devices such as a keyboard and a mouse, and output devices such as a display, and transfers information input by the operator via the input devices to the calculation unit 31 or the UI 39, and displays images on the display in response to commands from the calculation unit 31.

[0031] Figure 4A is a flowchart showing an example of bin picking performed in the part grasping system of Figure 1, Figure 4B is a flowchart showing an example of patch image processing performed in the bin picking of Figure 4A, Figure 4C is a flowchart showing an example of grasping inference performed in the bin picking of Figure 4A, and Figure 4D is a flowchart showing an example of determining a part to be grasped performed in the grasping inference of Figure 4C.

[0032] In step S101 of bin picking in Fig. 4A, a plan view image of a large number of parts stacked in the storage compartment 911 of the parts bin 91 is captured by the camera 81. As described above, a grayscale image Ig and a depth image Id are captured as plan view images. The main control unit 311 transfers these images Id and Ig acquired from the camera 81 to the image synthesis unit 41 of the image processing unit 4, and the image synthesis unit 41 performs patch image processing (step S102).

[0033] Figures 5A to 5E are diagrams schematically showing operations executed in the patch image processing of Figure 4B. In step S201 of the patch image processing of Figure 4B, the image synthesis unit 41 generates a synthesized image Ic (Figure 5C) by synthesizing a grayscale image Ig (Figure 5A) and a depth image Id (Figure 5B).

[0034] As shown in Fig. 5A, the grayscale image Ig is composed of a plurality of pixels PX arranged two-dimensionally in the X and Y directions, and is image data indicating the luminance Vg of each of the plurality of pixels PX. Note that in Fig. 5A, a pixel PX is identified by a combination (m,n) of "m" indicating the row number and "n" indicating the column number, and a pixel PX(m,n) of the grayscale image Ig has a luminance Vg(m,n). Note that the luminance Vg(m,n) has a larger value as the corresponding location becomes brighter.

[0035] As shown in Fig. 5B, the depth image Id is composed of multiple pixels PX, similar to the grayscale image Ig, and is image data indicating the depth (distance) of each pixel PX. The same notation as in Fig. 5A is used in Fig. 5B, and a pixel PX(m,n) in the depth image Id has a depth Vd(m,n). The shallower the depth of the corresponding location (in other words, the higher the position of the opposing location), the greater the value of the depth Vd(m,n).

[0036] As shown in Fig. 5C, the composite image Ic is composed of a plurality of pixels PX, similar to the grayscale image Ig, and is image data indicating a composite value Vc of the pixel PX for each of the plurality of pixels PX. In Fig. 5C, the same notation as in Fig. 5A is used, and a pixel PX(m,n) of the composite image Ic has a composite value Vc(m,n).

[0037] The composite value Vc(m,n) is calculated by the following equation: Vc(m,n)=Vd(m,n)×(1+Vg(m,n) / max(Vg)) max(Vg) is the maximum luminance value Vg contained in the grayscale image Ig. That is, the composite value Vc is the brightness Vg weighted by the depth Vd, and the composite image Ic is a depth-weighted grayscale image. In the above formula, the brightness Vg normalized by the maximum brightness is multiplied by the depth Vd (weight). However, normalization is not essential, and the composite value Vc may be calculated by multiplying the brightness Vg as is by the depth Vd (weight). In short, it is sufficient to determine the composite value Vc so that it depends on both the brightness Vg and the depth Vd.

[0038] FIG. 5D shows the experimental results of generating a composite image Ic from a grayscale image Ig and a depth image Id. The grayscale image Ig (before filtering) is two-dimensional image data acquired by the camera 81, and the grayscale image Ig (after filtering) is two-dimensional image data from which predetermined components (high-frequency components) have been removed by filtering. The depth image Id (before filtering) is three-dimensional image data acquired by the camera 81, and the depth image Id (before filtering) is three-dimensional image data from which predetermined components (high-frequency components) have been removed by filtering. The composite image Ic is a depth-weighted grayscale image obtained by combining the filtered grayscale image Ig and the depth image Id using the above formula. Looking at the ranges (elliptical ranges) indicated by the arrows in the "Grayscale Image Ig (after filtering)" and "Composite Image Ic" columns, we see that components that were clearly visible in the grayscale image Ig (after filtering) are not visible in the composite image Ic. This is because the corresponding part is deep (in other words, low in height) and a small weight is assigned to the brightness Vg of the corresponding part. In this way, combining the grayscale image Ig and the depth image Id has the effect of highlighting the parts at higher positions. Note that the filter used in FIG. 5D is not essential, and the same effect can be obtained even if it is omitted as appropriate.

[0039] The composite image Ic generated in step S201 of FIG. 4B is output from the image composition unit 41 to the patch image generation unit 43, and the patch image generation unit 43 performs image processing of steps S202 to S204 on the composite image Ic. Specific details of this image processing are illustrated in FIG. 5E. In step S202, the composite image Ic is binarized using a predetermined threshold value to obtain a binary composite image Ic. In this binary composite image Ic, closed regions with high luminance (white) appear corresponding to components; in other words, the closed regions in the binary composite image Ic can be recognized as components P. In step S203, the patch image generation unit 43 performs labeling to associate different labels (numbers) with each component P (closed region Rc) in the binary composite image Ic.

[0040] In step S204, a cropping range Rc is set for cropping an image including the part P from the binary composite image Ic. In particular, the cropping range Rc is set to represent the position of the robot hand 51 when grasping the part P. This cropping range Rc corresponds to the range (grasping target range) that the robot hand 51 targets for grasping, and the robot hand 51 can grasp the part P present in the cropping range Rc. For example, in the "patch image Ip" column in FIG. 5E, the portions of the cropping range Rc corresponding to the two claws 511 of the robot hand 51 that face the part P(2) from above to grasp the part P are represented by white solid lines (parallel to the Y direction), and the trajectories of the movement of both ends of each claw 511 are represented by white dashed lines (parallel to the X direction). As can be seen from this example, the claws 511 are parallel to the Y direction, and the rotation angle of the robot hand 51 in the θ direction is zero degrees. In other words, the cropping range Rc is set when the rotation angle of the robot hand 51 in the θ direction is zero degrees. Then, the patch image generating unit 43 acquires the image of the cutout range Rc from the binary composite image Ic as a patch image Ip (patch image generation). This patch image Ip is generated for each part P labeled in step S203.

[0041] As shown in FIG. 4A, once the patch image processing in step S102 is completed, grasp inference in step S103 (FIG. 4C) is executed. FIGS. 6A to 6C and 7 are diagrams schematically showing the operations executed in the grasp inference in FIG. 4C. To start the grasp inference in FIG. 4C, patch image information (FIG. 6A) indicating the multiple patch images Ip acquired in the patch image processing in step S102 is output from the image synthesis unit 41 to the alignment network unit 45. As shown in FIG. 6A, the patch image information indicates the patch image Ip, the label number of the patch image Ip, and the position of the cutout range Rc of the patch image Ip in association with each other. The shape of the cutout range Rc is the same for each patch image Ip, and the position (cutout position) of the cutout range Rc is specified by the X coordinate, Y coordinate, and θ coordinate of the geometric center of gravity of the cutout range Rc.

[0042] In contrast, in step S301 of FIG. 4C, the alignment network unit 45 resets the count value for counting the labels of the multiple patch images Ip indicated by the patch image information to zero (step S301), and increments the count value (step S302).

[0043] In step S303, the alignment network unit 45 determines whether the area of ​​the object (white closed region) included in the patch image Ip of the current count value is appropriate. Specifically, the object area is compared with a lower threshold and an upper threshold that is greater than the lower threshold. If the object area is smaller than the lower threshold or greater than the upper threshold, the object area is determined to be inappropriate ("NO" in step S303), and the process returns to step S302. On the other hand, if the object area is equal to or greater than the lower threshold and equal to or less than the upper threshold, the object area is determined to be appropriate ("YES" in step S303), and the process proceeds to step S304.

[0044] In step S304, the alignment network unit 45 calculates the amount of correction to correct the position of the cutout range Rc based on the patch image Ip of the current count value. That is, the alignment network unit 45 has an alignment neural network, which outputs the amount of correction (Δx, Δy, Δθ) for the cutout range Rc when the patch image Ip is input. The relationship between the patch image Ip and the amount of correction for the cutout range Rc will be described with reference to FIG. 6C.

[0045] The "Cut-out range Rc" column in FIG. 6C shows the cut-out range Rc and the patch image Ip cut out from the cut-out range Rc, and the "Corrected cut-out range Rcc" column shows the corrected cut-out range Rcc obtained by correcting the position of the cut-out range Rc according to the correction amount (Δx, Δy, Δθ) superimposed on the cut-out range Rc and the patch image Ip cut out from the cut-out range Rc. The cut-out range Rc and the corrected cut-out range Rcc have the same shape, and the following operations are performed: ·Translate in the X direction by a correction distance Δx...X direction parallel operation · Parallel movement in the Y direction by the correction distance Δy...Y direction parallel operation Rotation in the θ direction by the correction angle Δθ...Rotation in the θ direction The cutout range Rc after this correction is identical to the corrected cutout range Rcc. Furthermore, the misalignment between the center of the corrected cutout range Rcc and the component P is improved compared to the misalignment between the center of the cutout range Rc and the component P. In other words, the correction of the cutout range Rc is a correction that reduces the misalignment between the cutout range Rc and the component P; more specifically, a correction that converts the cutout range Rc into a corrected cutout range Rcc so that the component P is centered. The alignment neural network of the alignment network unit 45 receives a patch image Ip and outputs a correction amount (Δx, Δy, Δθ) for correcting the cutout range Rc of the patch image Ip and calculating the corrected cutout range Rcc. The calculation of correcting the cutout range Rc by the correction amount and converting it into the corrected cutout range Rcc can be performed by multiplying a rotation matrix that rotates the cutout range Rc by Δθ in the θ direction by a translation matrix that translates the cutout range Rc by Δx in the X direction and Δy in the Y direction. If image scaling needs to be taken into consideration, a scaling matrix can be further multiplied.

[0046] 6C, when the part P has a shape that is long in a predetermined direction, it is preferable to center the part P so that the long axis direction of the part P is perpendicular to the gripping direction G of the robot hand 51. This allows the part P to be gripped accurately by the robot hand 51.

[0047] In step S305, the alignment network unit 45 creates a corrected cut-out range Rcc by correcting the cut-out range Rc based on the correction amount output by the alignment neural network, and acquires the image of the corrected cut-out range Rcc from the binary composite image Ic as a corrected patch image Ipc (corrected patch image generation). Steps S302 to S305 are then repeated for all labels included in the patch image information (in other words, all patch images Ip) until they are completed (until step S306 returns "YES").

[0048] When the process is completed for all labels, correction patch image information ( FIG. 6B ) indicating multiple correction patch images Ipc is output from the alignment network unit 45 to the grip classification network unit 47. As shown in FIG. 6B , the correction patch image information indicates the correction patch image Ipc, the label number of the correction patch image Ipc, and the position of the correction cut-out range Rcc of the correction patch image Ipc, in association with each other. The shape of the correction cut-out range Rcc is the same for each correction patch image Ipc, and the position (cut-out position) of the correction cut-out range Rcc is specified by the X-coordinate, Y-coordinate, and θ-coordinate of the geometric center of gravity of the correction cut-out range Rcc.

[0049] In step S307, the grip classification network unit 47 calculates a grip success probability for each of the multiple corrected patch images Ipc indicated in the corrected patch image information. Specifically, the grip success probability (grasp success probability) is calculated when an attempt is made to grip the part P indicated by the corrected patch image Ipc cut out in the corrected clipping range Rcc with the robot hand 51 positioned at position (x+Δx, y+Δy, θ+Δθ) of the corrected clipping range Rcc. That is, the grip classification network unit 47 has a grip classification neural network, which, when inputted with the corrected patch image Ipc, outputs the grip success probability corresponding to the corrected patch image Ipc. In this way, the grip success probability information shown in FIG. 7 is acquired. As shown in FIG. 7, the grip success probability information indicates the corrected patch image Ip, the label number of the corrected patch image Ipc, the position of the corrected clipping range Rcc of the corrected patch image Ipc, and the grip success probability of the corrected patch image Ipc, in association with each other. Although the probability of successful grasping is shown as a value between 0 and 1 in the example of FIG. 7, it may also be shown as a percentage.

[0050] In step S308, the main control unit 311 determines the part P to be grasped based on the grasping success probability information output from the grasping classification network unit 47. In determining the part to be grasped in Fig. 4D, each correction patch image Ipc of the grasping success probability information is sorted in descending order according to the grasping success probability (step S401). In other words, the correction patch image Ipc with a higher grasping success probability is sorted higher.

[0051] Furthermore, correction patch images Ipc with the same grasping success probability are sorted in descending order according to the object area included in the correction patch image Ipc. That is, the larger the object area of ​​a correction patch image Ipc, the higher it is sorted. In step S403, the count value of the sorting order is reset to zero, and in step S404, the count value is incremented.

[0052] In step S405, it is determined whether the part P included in the correction patch image Ipc of the current count value is close to the edge of the storage compartment 911 (container) of the part bin 91. Specifically, if the distance between the position of the correction cropping range Rcc from which the correction patch image Ipc is cropped and the wall surface of the storage compartment 911 is less than a predetermined value, it is determined that the part P is close to the edge of the container ("YES" in step S405), and the process returns to step S404. On the other hand, if the distance is equal to or greater than a predetermined value, it is determined that the part P is not close to the edge of the container ("NO" in step S405), and the process proceeds to step S406. In step S406, the correction patch image Ipc of the current count value is selected as one correction patch image Ipc representing the part P to be grasped. The process then returns to the flowchart of FIG. 4A.

[0053] 4A, the robot hand 51 moves to the position indicated by the correction cropping range Rcc corresponding to the one correction patch image Ipc selected in step S103, and grasps the part P indicated by the one correction patch image Ipc. In step S105, the camera 83 captures an image of the part P grasped by the robot hand 51, and in step S106, the main control unit 311 determines the part P grasped by the robot hand 51 from the image captured by the camera 83. Furthermore, the main control unit 311 determines whether the number of grasped parts P is one (step S107), and if it is not one ("NO" in step S107), it causes the robot hand 51 to return the part P to the storage compartment 911 of the part bin 91 (step S108). Furthermore, if the number of grasped parts P is one (if "YES" in step S107), the main control unit 311 determines whether the grasped part P is normal (step S109), and if there is an abnormality such as the area representing the part P being too small (if "NO" in step S109), it causes the robot hand 51 to discard the part P in the trash can 93 (step S110).

[0054] On the other hand, if the part P is normal ("YES" in step S109), the main control unit 311 causes the robot hand 51 to place the part P in the storage compartment 921 of the kitting tray 92 (step S111). Next, the main control unit 311 measures the mass using the mass meter 85 (step S112) and determines whether the mass indicated by the mass meter 85 is appropriate (step S113). Specifically, this determination can be made based on whether the mass corresponding to the part P placed on the kitting tray 92 has increased. If the mass is not appropriate ("NO" in step S113), the main control unit 311 notifies the operator of the abnormality using the UI 39. On the other hand, if the mass is appropriate ("YES" in step S113), the main control unit 311 returns to step S101.

[0055] The above is the content of bin picking executed by the part holding system 1. In the above-described holding inference, the alignment network unit 45 calculates the correction amount (Δx, Δy, Δθ) for correcting the cut-out range Rc based on the patch image Ip cut out from the cut-out range Rc. In particular, the alignment network unit 45 calculates the correction amount for the cut-out range Rc from the patch image Ip using an alignment neural network. Next, a method for making this alignment neural network learn the relationship between the patch image Ip and the correction amount for the cut-out range Rc will be described.

[0056] 8A is a flowchart showing an example of a method for collecting learning data for an alignment neural network. This flowchart is executed by the calculation unit 31 of the control device 3. When executing this flowchart, a simulator that performs bin picking in a part gripping system 1 virtually constructed by calculation (hereinafter referred to as the "virtual part gripping system 1") is configured in the calculation unit 31. This simulator virtually performs, by calculation, the operation of the robot hand 51 to grip a part P from the storage compartment 911 of the part bin 91, based on physical parameters such as gravitational acceleration and friction coefficient.

[0057] In step S501, it is confirmed whether the number of data required for learning has been acquired. This required number can be set in advance by, for example, an operator. If the required number of data has been acquired (YES in step S501), the flow chart of FIG. 8A ends. If the number of acquired data is less than the required number (NO in step S501), the flow proceeds to step S502.

[0058] In step S502, it is determined whether a sufficient number of parts P are stored in the storage compartments 911 of the part bin 91 arranged in the virtual part holding system 1. Specifically, this determination can be made based on whether the number of parts P is equal to or greater than a predetermined number. If the number of parts P in the storage compartments 911 of the part bin 91 is less than the predetermined number (if "NO" in step S502), the number of parts P in the storage compartments 911 of the part bin 91 is reset to the initial value (step S503), and the process returns to step S501. On the other hand, if the number of parts P in the storage compartments 911 of the part bin 91 is equal to or greater than the predetermined number (if "YES" in step S502), the process proceeds to step S504.

[0059] In step S504, a composite image Ic is generated in the virtual part holding system 1 in the same manner as in the above-described actual part holding system 1. Next, this composite image Ic is binarized to generate a binary composite image Ic, and each part P included in this binary composite image Ic is labeled (step S505). Then, a cutout range Rc is set for each labeled part P, and a patch image Ip is cut out (step S506).

[0060] In step S507, the count value for counting each patch image Ip is reset, and in step S508, the count value is incremented. Then, in the same manner as above, it is determined whether the area of ​​the object (white closed region) included in the patch image Ip of the current count value is appropriate (step S509). If the area of ​​the object is inappropriate (if "NO" in step S509), the process returns to step S508, whereas if the area of ​​the object is appropriate (if "YES" in step S509), the process proceeds to step S510.

[0061] Once one patch image Ip with an appropriate object area is selected, the main control unit 311 generates a position determination mask Mp ( FIG. 8B ) from the selected patch image Ip (step S510). FIG. 8B is a diagram schematically illustrating an example of a position determination mask generated from a patch image. This position determination mask Mp has an outline with the same shape as the patch image Ip (in other words, the cropping range Rc), and a component reference pattern Pr with the same shape as the component P included in the patch image Ip is placed in the center of the position determination mask Mp. This component reference pattern Pr is generated so as to have the same number of pixels in both the vertical and horizontal directions as the component P (in other words, the closed white region) included in the patch image Ip. This position determination mask Mp is a model of an ideal patch image Ip in which the component P is located at the center. The patch image Ip and the position determination mask Mp generated from the selected patch image Ip are then associated with each other and stored in a patch image list (step S511).

[0062] Once each step up to step S511 is completed in this manner, the process returns to step S501. Steps S501 to S511 are then repeatedly executed until the required number of data is acquired, in other words, until the number of pairs of patch images Ip and position determination masks Mp stored in the patch image list reaches the required number.

[0063] 9A is an example of a flowchart for causing the alignment neural network to learn the training data collected in FIG. 8A. This flowchart is executed by the calculation unit 31 of the control device 3. In step S601, it is determined whether the number of learning operations has reached a predetermined number. This predetermined number can be set in advance by, for example, an operator.

[0064] In step S602, an untrained patch image Ip selected from the patch image list is forward propagated to the alignment neural network of the alignment network unit 45. As a result, a correction amount (Δx, Δy, Δθ) corresponding to the patch image Ip is output from the neural network of the alignment network unit 45. Furthermore, the alignment network unit 45 generates a corrected patch image Ipc by cutting out the binary composite image Ic (generated in step S505) within the corrected cut-out range Rcc obtained by correcting the cut-out range Rc using this correction amount (step S603).

[0065] In step S604, the alignment network unit 45 overlays the position determination mask Mp corresponding to the patch image Ip selected in step S602 and the corrected patch image Ipc so that their contours match, and calculates the mean square error between the component reference pattern Pr of the position determination mask Mp and the component P included in the corrected patch image Ipc as a loss function. Then, in step S605, the loss function is backpropagated in the alignment neural network (error backpropagation) to update the parameters of the alignment neural network.

[0066] Note that the loss function can be calculated without using the position determination mask Mp. That is, the principal axis angle may be calculated from the image moments of the part P, and the mean square error between this principal axis angle and a predetermined reference angle may be used as the loss function. On the other hand, in the example shown in FIG. 9B, it is advantageous to use the position determination mask Mp. FIG. 9B is a diagram schematically illustrating an example in which the use of a mask is advantageous for calculating the loss function. The part P included in the correction patch image Ipc shown in FIG. 9B has a zigzag shape, and it is difficult to appropriately determine the principal axis angle from the image moments of this part P. Therefore, the position determination mask Mp is used here from the perspective of being able to accommodate parts P of various shapes.

[0067] In step S606, patch images Ip (test data) reserved for testing and not used for learning among the patch images Ip stored in the patch image list are forward propagated through the alignment neural network with updated parameters to calculate a correction amount. Then, based on this correction amount, a loss function is calculated using the position determination mask Mp corresponding to the test data in the same manner as in steps S603 and S604 above.

[0068] Each time step S606 is executed, the calculation unit 31 stores the loss function calculated in step S606 and calculates the minimum value among the stored loss functions. The calculation unit 31 then checks whether the most recently calculated loss function has updated the minimum value. In particular, step S607 determines whether the minimum value has not been updated for 10 consecutive times, i.e., whether a loss function greater than the minimum value has been calculated. If a loss function equal to or less than the minimum value has been calculated in the past 10 times (if "NO" in step S607), the process returns to step S601. On the other hand, if a loss function greater than the minimum value has been calculated for 10 consecutive times (if "YES" in step S607), the flowchart of FIG. 9A ends. The number of times is not limited to 10 and can be changed as needed.

[0069] In the above-described grasp inference, when the corrected patch image Ipc is input to the grasp classification network unit 47, the grasp classification network unit 47 calculates the grasp success probability when the part P included in the corrected patch image Ipc is grasped by the robot hand 51 at the position indicated by the corrected patch image Ipc. In particular, the grasp classification network unit 47 calculates the grasp success probability from the corrected patch image Ipc using a grasp classification neural network. Next, a method for making this grasp classification neural network learn the relationship between the corrected patch image Ipc and the grasp success probability will be described.

[0070] 10A to 10C are an example of a flowchart for causing the grip classification neural network to perform learning. This flowchart is executed by the calculation unit 31 of the control device 3. As with the learning of the above-described alignment neural network, the learning of the grip classification neural network also uses a simulator that constructs the virtual part grip system 1.

[0071] In the flowchart of Fig. 10A, learning data is collected in the same manner as in Fig. 8A. That is, steps S701 to S709 in Fig. 10A are the same as steps S501 to S509 in Fig. 8A, except for the following point. That is, in step S701, it is determined whether the number of learning operations has reached a predetermined number, rather than the number of acquired data. This predetermined number can be set in advance by, for example, an operator.

[0072] 10A, when one patch image Ip having an appropriate object area is selected by executing steps S701 to S709, alignment network unit 45 calculates a correction amount corresponding to patch image Ip using the alignment neural network that has completed the above learning (step S710), and stores the patch image Ip and the correction amount in association with each other in the correction amount list (step S711). Steps S708 to S711 are then repeated until the count value reaches a maximum (until step S712 returns "YES"), and pairs of patch image Ip and correction amounts are sequentially stored in the correction amount list. When the count value reaches a maximum (step S712 returns "YES"), the process proceeds to step S712 in FIG. 10B.

[0073] In step S712, the alignment network unit 45 corrects the cut-out range Rc of the patch image Ip based on the correction amount to generate a corrected cut-out range Rcc, and executes the process of generating a corrected patch image Ipc based on the corrected cut-out range Rcc for each pair of patch image Ip and correction amount stored in the correction amount list. In this way, multiple correction patch images Ipc are generated. The specific procedure for generating the correction patch image Ipc is as described above.

[0074] In step S713, it is confirmed whether the number of data required for learning has been acquired. This required number can be set in advance by, for example, an operator. If the required number of data has been acquired ("YES" in step S713), the process proceeds to step S717 (FIG. 10C) described below. If the number of acquired data is less than the required number ("NO" in step S713), the process proceeds to step S714.

[0075] In step S714, one correction patch image Ipc is selected randomly (for example, based on the output of a random number generator) from the multiple correction patch images Ipc generated in step S712. Then, in step S715, in the virtual part gripping system 1, gripping of the part P included in the one correction patch image Ipc is attempted by the robot hand 51 located at the position of the one correction patch image Ipc. Note that the position of the correction patch image Ipc corresponds to the position of the correction cropping range Rcc into which the correction patch image Ipc is cropped. Then, the success or failure result of the gripping attempt (1 if successful, 0 if unsuccessful) is stored in a success or failure result list in association with the one correction patch image Ipc (step S716), and the process returns to step S701 in FIG. 10A.

[0076] On the other hand, if it is determined in step S713 that the required amount of data has been acquired (YES), the process proceeds to step S717 in FIG. 10C as described above. In step S717, a horizontally flipped correction patch image Ipc obtained by flipping the correction patch image Ipc horizontally, a vertically flipped correction patch image Ipc obtained by flipping the correction patch image Ipc vertically, and a vertically flipped correction patch image Ipc obtained by flipping the correction patch image Ipc horizontally and vertically are generated. As a result, three types of images, the horizontally flipped correction patch image Ipc, the vertically flipped correction patch image Ipc, and the vertically flipped correction patch image Ipc, are prepared for each correction patch image Ipc in the success / failure result list. In other words, a number of correction patch images Ipc equal to three times the number of correction patch images Ipc stored in the success / failure result list are prepared.

[0077] In step S718, each of the multiple correction patch images Ipc generated in step S717 is forward propagated through the grasping classification neural network of the grasping classification network unit 47, and a grasping success probability is calculated for each correction patch image Ipc. Then, in step S719, an average value of the grasping success probability for each of the horizontally flipped correction patch image Ipc, the vertically flipped correction patch image Ipc, and the vertically and horizontally flipped correction patch image Ipc generated from the same correction patch image Ipc is calculated. As a result, an average value of the grasping success probability is calculated for each correction patch image Ipc to be stored in the success / failure result list.

[0078] In step S720, a random number generator generates one of the values ​​"0," "1," and "2." If the random number generation results in "0," one correction patch image Ipc is randomly selected from among the correction patch images Ipc whose grasping success probabilities were calculated in step S719 (step S721). If the random number generation results in "1," one correction patch image Ipc whose grasping success probability is closest to "0.5" (in other words, 50%) is selected from among the correction patch images Ipc (step S722). If the random number generation results in "2," one correction patch image Ipc whose grasping success probability is highest is selected from among the correction patch images Ipc (step S723).

[0079] In step S724, the virtual part gripping system 1 attempts to grip the part P indicated by one of the correction patch images Ipc using the robot hand 51 located at the position of that one of the correction patch images Ipc. Then, a loss function is calculated based on the result of whether the part gripping attempt was successful (1 if successful, 0 if unsuccessful) and the average gripping success probability calculated for that one of the correction patch images Ipc in step S719. As the loss function, various well-known functions such as cross-entropy error can be used.

[0080] Furthermore, the calculation unit 31 stores the loss function calculated in step S725 each time step S725 is executed, and calculates the minimum value among the stored loss functions. The calculation unit 31 then checks whether the most recently calculated loss function has updated the minimum value. In particular, step S726 determines whether the minimum value has not been updated for 10 consecutive times, i.e., whether a loss function greater than the minimum value has been calculated. If a loss function equal to or less than the minimum value has been calculated in the past 10 times (if "NO" in step S726), the grasp success / failure result of step S724 is stored in a success / failure result list in association with one correction patch image Ipc (step S727). Then, in step S728, the parameters of the grasp classification neural network are updated by backpropagating the loss function calculated in step S725 through the grasp classification neural network (error backpropagation). On the other hand, if a loss function greater than the minimum value is calculated ten consecutive times in step S726 (if "NO"), the process returns to step S701 in Fig. 10A. Note that the number of times is not limited to ten and can be changed as needed.

[0081] In the embodiment described above, a composite image Ic is generated (step S201) by combining grayscale images Ig (brightness images) and depth images Id that respectively represent multiple parts P. In the composite image Ic generated in this manner, the shapes of parts P that are located at relatively high positions among the multiple parts P tend to remain, and this is useful for recognizing such parts P.

[0082] Furthermore, based on the result of recognizing the part P in this way, the robot hand 51 grasps the part P (step S104). Therefore, it becomes possible for the robot hand 51 to grasp the part with a high success rate.

[0083] Furthermore, the grayscale image Ig indicates the brightness Vg for each pixel PX, the depth image Id indicates the depth Vd for each pixel PX, and the composite image Ic indicates the composite value Vc for each pixel PX. In particular, the image composition unit 41 generates the composite image Ic by calculating the composite value Vc in which the brightness Vg is weighted by the depth Vd. The composite image Ic generated in this manner tends to retain the shapes of components P at relatively high positions among multiple components P, and is useful for recognizing such components P.

[0084] Furthermore, the image synthesis unit 41 uses the following formula Vc = Vd × (1 + Vg / max(Vg)) max(Vg) is the maximum value of the luminance Vg contained in the grayscale image Ig. The composite image Ic thus generated is useful for recognizing parts P because it is easy to retain the shapes of parts P that are located at relatively high positions among the multiple parts P.

[0085] Furthermore, a corrected cutout range Rcc (hand range) indicating the range of the robot hand 51 is set for the component P shown in the composite image IC, and a corrected patch image Ipc (target component image) indicating the component P included in the corrected cutout range Rcc is cut out from the composite image Ic, thereby generating multiple corrected patch images Ipc each indicating a different component P (steps S301 to S306). Furthermore, the probability of successful grasping when the robot hand 51 attempts to grasp the component P based on the positional relationship between the robot hand 51 and the component P indicated by the corrected patch image Ipc is calculated for each of the multiple corrected patch images Ipc (step S307). As described above, the composite image Ic used at this time is likely to retain the shape of a component P located at a relatively high position. Therefore, it is possible to reliably generate a corrected patch image Ipc that includes a component P located at a relatively high position, in other words, a component P with a relatively high probability of successful grasping.

[0086] Furthermore, one correction patch image Ipc is selected from the plurality of correction patch images Ipc based on the grasp success probability calculated for each of the plurality of correction patch images Ipc, and it is determined that the part P indicated by the one correction patch image Ipc will be grasped by the robot hand 51 in the posture indicated by the one correction patch image Ipc (steps S401 to S406). With this configuration, it is possible to reduce the frequency of failure in grasping the part P by the robot hand 51.

[0087] Furthermore, among the multiple correction patch images Ipc that do not satisfy the predetermined exclusion condition (step S405), the correction patch image Ipc with the highest probability of successful grasping is selected as one correction patch image Ipc (steps S401 to S406). With this configuration, it is possible to reliably reduce the frequency of failure in grasping the part P by the robot hand 51.

[0088] Specifically, the predetermined exclusion condition is that the distance between the edge of the compartment storage 911 (container) of the part bin 91 and the position of the correction patch image Ipc is less than a predetermined value (step S406). This makes it possible to prevent a situation in which gripping of the part P fails due to the influence of the wall surface at the edge of the compartment storage 911.

[0089] The grip classification network unit 47 also has a grip classification neural network that has learned the relationship between the corrected patch image Ipc and the grip success probability using as training data the success or failure of gripping when the robot hand 51 attempts to grip the part P in the positional relationship between the robot hand 51 and the part P indicated by the corrected patch image Ipc. The grip classification neural network then calculates the grip success probability for each of the multiple corrected patch images Ipc. With this configuration, it is possible to reliably reduce the frequency of failure to grip the part P by the robot hand 51 by using the learning results of the grip classification neural network.

[0090] Furthermore, this embodiment can be considered suitable for learning using such a grasp classification neural network. That is, one example of a method for training a grasp classification neural network is to input both a grayscale image Ig and a depth image Id to the grasp classification neural network and have the grasp classification neural network learn the relationship between these images Ig and Id and the grasp success probability. However, learning the grasp success probability of the robot hand 51 from the association between the brightness of the part P indicated by the grayscale image Ig and the height of the part P indicated by the depth image Id is difficult for the grasp classification neural network and requires a large computational load. On the other hand, in this embodiment, a composite image Ic (corrected patch image Ipc) obtained by combining the grayscale image Ig and the depth image Id is input to the grasp classification neural network, and the grasp classification neural network can learn based on this composite image Ic. As a result, learning can be performed relatively easily and the computational load can be reduced.

[0091] Furthermore, gripping state information (a side-view image and mass of the part P) indicating the state in which the part P indicated by one of the correction patch images Ipc is actually gripped by the robot hand 51 in the posture indicated by that one of the correction patch images Ipc is acquired (steps S105, S112). Then, based on this gripping state information, it is determined whether the robot hand 51 has successfully gripped the part P (steps S107, S109, S113). With this configuration, it is possible to confirm whether the selected part P has actually been successfully gripped based on the gripping success probability calculated for the correction patch image Ipc.

[0092] As described above, in the above embodiment, the part gripping system 1 corresponds to an example of a "part gripping system" of the present invention, the control device 3 corresponds to an example of an "image processing device" of the present invention, the main control unit 311 corresponds to an example of an "image acquisition unit" and a "success / failure determination unit" of the present invention, the image synthesis unit 41 corresponds to an example of an "image synthesis unit" of the present invention, the patch image generation unit 43, the alignment network unit 45, and the grip classification network unit 47 work together to function as an example of a "success probability calculation unit" of the present invention, the robot hand 51 corresponds to an example of a "robot hand" of the present invention, the camera 83 corresponds to an example of a "camera" of the present invention, the compartment storage 911 of the part bin 91 corresponds to an example of a "container" of the present invention, the depth image Id corresponds to an example of a "depth image" of the present invention, the grayscale image Ig corresponds to an example of a "brightness image" of the present invention, the composite image Ic corresponds to an example of a "composite image" of the present invention, the corrected patch image Ipc corresponds to an example of a "target part image" of the present invention, and the corrected cut-out range Rcc corresponds to an example of a "hand range" of the present invention.

[0093] The present invention is not limited to the above-described embodiment, and various modifications can be made to the above-described embodiment without departing from the spirit of the present invention. For example, in step S105, a plurality of side-view images of the part P held by the robot hand 51 may be acquired by using the camera 83 to capture images of the part P from different directions. These side-view images can be acquired, for example, by capturing images of the part P while rotating the robot hand 51 holding the part P in the θ direction. This allows the number of parts P to be confirmed in step S107 and any abnormalities in the part P (such as an area that is too small) to be confirmed from multiple directions in step S109.

[0094] Furthermore, the learning of the grip classification neural network may be performed by executing the flowchart of Fig. 11. Here, Fig. 11 is a flowchart showing an example of a method for re-learning the grip classification neural network of the grip classification network unit. This flowchart is executed by the main control unit 311, for example, when the planned bin picking is completed.

[0095] In step S801, the main control unit 311 checks the history of past bin picking operations for abnormalities detected based on side-view images ("NO" in steps S107 and S108) or mass measurements ("NO" in step S113). If the number of times an abnormality has been detected is equal to or greater than a predetermined number ("YES" in step S802), the grip classification neural network of the grip classification network unit 47 is retrained (step S803). In this retraining, the corrected patch image Ipc representing the part P for which an abnormality was detected and the grip success / failure result (i.e., failure) are used as training data. Specifically, an error function is calculated based on the grip success / failure probability and the grip success / failure result (failure) obtained by forward propagating the corrected patch image Ipc through the grip classification neural network. Then, this error function is backpropagated in the grip classification neural network to update the parameters of the grip classification neural network (retraining).

[0096] 11, the grip classification neural network is made to re-learn based on the results of acquiring grip state information (side-view image, mass) for the part P gripped by the robot hand 51. In this configuration, the grip classification neural network is re-learned in accordance with the actual success or failure of gripping the part P selected based on the grip success probability calculated for the corrected patch image Ipc, thereby improving the accuracy of the calculation of the grip success probability by the grip classification neural network.

[0097] 12 shows a modified example of the grasp classification neural network of the grasp classification network section. In this grasp classification neural network 471, a multi-layer convolutional neural network 472 and a fully connected layer 473 are arranged in series. In addition, a spatial attention module 474 and a channel attention module 475 are provided on the output side of each convolutional neural network 472. The feature map output from the convolutional neural network 472 is weighted by the spatial attention module 474 and the channel attention module 475 before being input to the next convolutional neural network 472 or the fully connected layer 473.

[0098] In particular, the attention mask Ma added to the feature map by the spatial attention module 474 has two attention regions Pg and Pp that pass through the center position of the corrected patch image Ipc (in other words, the corrected cropping range Rcc). In other words, the weights of the attention regions Pg and Pp in the attention mask Ma are greater than those of the other regions, and these weights are added to the feature map. Here, the attention region Pg is parallel to the gripping direction G, and the attention region Pp is perpendicular to the gripping direction G. In particular, when the long axis direction of the part P is perpendicular to the gripping direction G, as in the above example, the attention region Pp is parallel to the long axis direction of the part P. In other words, this attention mask Ma focuses on the attention region Pp corresponding to the ideal position of the part P in the corrected patch image Ipc and the attention region Pg corresponding to the path along which the claws 511 of the robot hand 51 approach the part P.

[0099] In the grip classification neural network, the attention mask Ma configured as described above is added to the feature map output from the convolutional neural network 472, and the feature map is weighted. Therefore, the angle of the long axis direction of the part P relative to the grip direction G and the status of the movement path of the robot hand 51 gripping the part P (whether or not other parts are present) can be accurately reflected in the judgment made by the grip classification neural network.

[0100] Furthermore, the method of generating the composite image Ic is not limited to the example using the above formula, and the composite image Ic may be generated using another formula that calculates the composite value Vc of the composite image Ic by weighting the brightness Vg of the grayscale image Ig by the depth Vd of the depth image Id.

[0101] In the above example, the grayscale image Ig and the depth image Id are combined to generate the composite image Ic. In this case, the composite image Ic may be generated by combining an inverted grayscale image Ig (luminance image) obtained by inverting the luminance of the grayscale image Ig with the depth image Id. In particular, when gripping a part P having a black-plated surface, it is preferable to generate the composite image Ic using the inverted grayscale image Ig.

[0102] Furthermore, it is not necessary to extract the patch image Ip from the binarized composite image Ic, and the patch image Ip may be extracted from the composite image Ic without performing binarization. The same applies to the correction patch image Ipc.

[0103] Furthermore, various modes for setting the cut-out range Rc for the part P in patch image processing are possible. For example, the cut-out range Rc can be set so that the geometric center of gravity of the cut-out range Rc coincides with the geometric center of gravity of the part P. However, this is not limiting, and in short, the cut-out range Rc may be set so as to include the target part P.

[0104] Furthermore, the specific configuration of the robot hand 51 is not limited to the above example. For example, the number of claws 511 of the robot hand 51 is not limited to two, and may be three or more. Also, a robot hand 51 that attracts by negative pressure or magnetic force may be used. Even in these cases, it is possible to set a cutout range Rc in the range to be grasped by the robot hand 51, and cut out the patch image Ip from the cutout range Rc.

[0105] Furthermore, in the above embodiment, the correction amount (Δx, Δy, Δθ) of the cut-out range Rc into which the patch image Ip is cut out is calculated by the positioning network unit 45, and the corrected patch image Ipc cut out in the corrected cut-out range Rcc obtained by correcting the cut-out range Rc by the correction amount (Δx, Δy, Δθ) is input to the grasping classification network unit 47. However, it is also possible to input the patch image Ip as it is to the grasping classification network unit 47 without performing correction using the correction amount (Δx, Δy, Δθ), and have the grasping classification network unit 47 calculate the grasping success probability directly from the patch image Ip. [Explanation of symbols]

[0106] 1...Part gripping system 3...Control device (image processing device) 311...Main control unit (image acquisition unit, success / failure determination unit) 41...Image synthesis unit 43...Patch image generation unit (success probability calculation unit) 45...Alignment network unit (success probability calculation unit) 47...Grasp classification network unit (success probability calculation unit) 51...Robot hand 83...Camera 91...Parts bin 911... Compartment storage (container) Id...depth image Ig...Grayscale image (brightness image) Ic...Composite image Ipc...Correction patch image (target part image) Rcc...Correction cutout range (hand range)

Claims

1. an image acquisition unit that acquires a brightness image showing a plurality of components housed in a container and a depth image showing the plurality of components; an image synthesis unit that generates a synthesized image by synthesizing the luminance image and the depth image acquired by the image acquisition unit; Equipped with the luminance image indicates a luminance Vg for each pixel, the depth image indicates a depth Vd for each pixel; The composite image indicates a composite value Vc for each pixel, The image synthesis unit generates the synthetic image by multiplying the brightness Vg by the depth Vd to calculate a synthetic value Vc in which the brightness Vg is weighted by the depth Vd, rather than extracting an area from the brightness image corresponding to an area in the depth image that is at a distance less than a predetermined threshold.

2. 2. The image processing device according to claim 1, further comprising a success probability calculation unit that generates a plurality of target part images each showing a different part by setting a hand range indicating the range of a robot hand for the part and creating target part images indicating the part included in the hand range based on the composite image, and calculates the probability of successful grasping for each of the plurality of target part images when the robot hand attempts to grasp the part in the positional relationship between the robot hand and the part indicated by the target part image.

3. 3. The image processing device according to claim 2, wherein the success probability calculation unit selects one target part image from the plurality of target part images based on the grasping success probability calculated for each of the plurality of target part images, and determines that the part indicated by the one target part image is to be grasped by the robot hand in the posture indicated by the one target part image.

4. The image processing device according to claim 3 , wherein the success probability calculation unit selects the target part image having the highest grasping success probability as the single target part image from among the plurality of target part images that do not satisfy a predetermined exclusion condition.

5. The image processing apparatus according to claim 4 , wherein the predetermined exclusion condition is that the distance between the edge of the container and the target part image is less than a predetermined value.

6. 6. The image processing device according to claim 3, wherein the success probability calculation unit has a neural network that has learned the relationship between the target part image and the grasping success probability using training data on whether the robot hand will be able to grasp the part when attempting to grasp the part in the positional relationship between the robot hand and the part shown in the target part image, and calculates the grasping success probability for each of the plurality of target part images using the neural network.

7. 7. The image processing device according to claim 6, further comprising a success / failure determination unit that acquires gripping state information indicating the state in which the part indicated by the one target part image is grasped by the robot hand in the posture indicated by the one target part image, and determines whether the part has been successfully grasped by the robot hand based on the gripping state information.

8. The image processing device according to claim 7 , wherein the success / failure determination unit acquires, as the gripping state information, a mass measured by a mass meter when the part gripped by the robot hand from the container is placed on the mass meter.

9. The image processing device according to claim 7 or 8, wherein the success / failure determining unit acquires, as the gripping state information, an image of the part gripped by the robot hand, captured by a camera.

10. The image processing device according to claim 9 , wherein the success / failure determining unit acquires, as the gripping state information, a plurality of images of the part gripped by the robot hand taken by the camera from different directions.

11. 11. The image processing device according to claim 7, wherein the success probability calculation unit causes the neural network to perform re-learning based on the results of acquiring the gripping state information for the part gripped by the robot hand.

12. An image processing device according to any one of claims 3 to 11; a robot hand that selects one target part image from the plurality of target part images, and grasps the part indicated by the one target part image based on a decision by the image processing device that the part indicated by the one target part image is to be grasped by the robot hand in a posture indicated by the one target part image; A part gripping system comprising:

13. acquiring a brightness image showing a plurality of components contained in a container and a depth image showing the plurality of components; generating a composite image by combining the luminance image and the depth image; Equipped with the luminance image indicates a luminance Vg for each pixel, the depth image indicates a depth Vd for each pixel; The composite image indicates a composite value Vc for each pixel, In the process of generating the composite image, the image processing method generates the composite image by calculating a composite value Vc in which the brightness Vg is weighted by the depth Vd by multiplying the brightness Vg by the depth Vd, rather than extracting an area from the brightness image corresponding to an area in the depth image that is at a distance less than a predetermined threshold.

14. acquiring a brightness image showing a plurality of components contained in a container and a depth image showing the plurality of components; generating a composite image by combining the luminance image and the depth image; a step of generating a plurality of target part images each showing a different part by setting a hand range indicating the range of a robot hand for the part and creating target part images indicating the part included in the hand range based on the composite image, and calculating a grasping success probability for each of the plurality of target part images when the robot hand attempts to grasp the part in the positional relationship between the robot hand and the part indicated by the target part image; selecting one target part image from the plurality of target part images based on the grasping success probability calculated for each of the plurality of target part images, and determining that the part indicated by the one target part image is to be grasped by the robot hand in the posture indicated by the one target part image; selecting one target part image from the plurality of target part images, and grasping the part indicated by the one target part image with the robot hand based on a determination that the part indicated by the one target part image should be grasped by the robot hand in the posture indicated by the one target part image; Equipped with the luminance image indicates a luminance Vg for each pixel, the depth image indicates a depth Vd for each pixel; The composite image indicates a composite value Vc for each pixel, In the process of generating the composite image, rather than extracting an area from the brightness image that corresponds to an area in the depth image that is at a distance less than a predetermined threshold, the composite image is generated by multiplying the brightness Vg by the depth vd to calculate a composite value Vc in which the brightness Vg is weighted by the depth Vd.

Images