Position correction device, robot system, and position correction program

Abstract

Provided is a position correction device capable of preventing the occurrence of erroneous detection or undetected problems and improving the recognition / detection accuracy of a workpiece. The position correction device calculates at least a detection position of at least one workpiece on the basis of a two-dimensional image obtained by capturing a presence region of a plurality of workpieces, and corrects and outputs the detection position of the at least one workpiece on the basis of three-dimensional measurement data obtained by measuring the presence region of the plurality of workpieces.

Description

Technical Field

[0001] The present disclosure relates to a position correction device, a robot system, and a position correction program. Background Art

[0002] Conventionally, there has been known an object detection technique for identifying an object (workpiece) in an image captured by a camera and obtaining information such as the position and orientation and the external dimensions of the object. In addition, there has also been known a method of extracting features from an image using image processing techniques (e.g., pattern matching) and identifying an object using the extracted features.

[0003] In addition, in recent years, a method of extracting features from an image using machine learning, identifying an object in the image, and estimating the position and orientation and the external dimensions of the object has also been put into practical use. In addition, a method of measuring the three-dimensional shape of an object using a three-dimensional measuring device and obtaining the position and orientation of the object using the measured three-dimensional measurement data (three-dimensional point cloud data) of the object has also been proposed.

[0004] Conventionally, various proposals have been made as an object detection technique for identifying an object in an image captured by a camera and obtaining information such as the position and orientation of the object.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2016-192135

[0008] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2010-120141 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] As described above, various proposals have been made as an object detection technique for identifying an object in an image captured by a camera and obtaining information such as the position and orientation of the object. In addition, in recent years, a method of estimating the position and orientation of an object based on features extracted from an image using machine learning has also been put into practical use, and a method of obtaining the position and orientation of an object using three-dimensional measurement data of the object obtained using a three-dimensional measuring device has also been proposed.

[0011] However, in the above-described conventional object detection techniques for obtaining information such as the position and orientation and the external dimensions of an object, for example, in actual applications, an object may be misidentified as multiple objects, or multiple objects may be misidentified as one object. In addition, when an object is close to its surrounding background, there are also problems of being unable to detect the object (undetected), or misidentifying the background as an object although there is actually no object.

[0012] Therefore, it is desirable to provide a position correction device, a robot system, and a position correction program that can prevent these false detections or undetected problems and improve the recognition / detection accuracy of workpieces (objects).

[0013] Means for Solving the Problem

[0014] According to an embodiment of the present invention, there is provided a position correction device that calculates at least the detection position of at least one workpiece based on a two-dimensional image obtained by photographing the presence areas of a plurality of workpieces, and corrects and outputs the detection position of at least one workpiece based on three-dimensional measurement data obtained by measuring the presence areas of the plurality of workpieces. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram schematically showing an example of the robot system of the present embodiment.

[0016] Figure 2 is a flowchart for explaining an example of the processing in the first embodiment of the position correction program of the present embodiment.

[0017] Figure 3 is a diagram for explaining an example of the identification process of the workpiece in the first embodiment of the position correction device of the present embodiment.

[0018] Figure 4 is a flowchart for explaining an example of the processing in the second embodiment of the position correction program of the present embodiment.

[0019] Figure 5 is a diagram for explaining an example of the identification process of the workpiece in the second embodiment of the position correction device of the present embodiment.

[0020] Figure 6 is a diagram for explaining the shape areas of the workpieces in the first and second embodiments of the position correction device of the present embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Hereinafter, embodiments of the position correction device, the robot system, and the position correction program of the present embodiment will be described in detail with reference to the drawings. In each drawing, the same or similar components are given the same or similar reference numerals. In addition, the embodiments described below do not limit the technical scope of the invention described in the claims and the meanings of the terms.

[0022] Figure 1 is a diagram schematically showing an example of the robot system of the present embodiment. As Figure 1As shown, the robot system 100 includes: a robot 1, a robot control device 2, a position correction device 3, and an acquisition unit (measurement unit) 4. The robot 1 includes: a robot mechanism unit 10, an arm 11, and an end effector (hand) 12.

[0023] The robot 1 is configured as a multi-axis robot, for example, and an end effector 12 is provided at the front end of the arm 11. In Figure 1 this case, the end effector 12 is an adsorption device (adsorption hand), which can of course be changed to various structures according to the object (workpiece) of the application robot system or the operation content, etc. In addition, the robot mechanism unit 10 makes the robot 1 perform a specified action according to a control instruction from the robot control device 2. That is, the robot control device 2 receives the output of the position correction device 3, and generates, for example, a control instruction for making the robot 1 perform a specified action according to a program or teaching data stored in an internal storage device in advance, and outputs it to the robot mechanism unit 10. In addition, although the position correction device 3 is described as being independent of the robot control device 2, it can also be configured to be assembled inside the robot control device 2.

[0024] The acquisition unit 4 acquires a two-dimensional image and three-dimensional measurement data (three-dimensional point cloud data) of the existence regions of a plurality of workpieces (for example, a plurality of corrugated cardboard boxes) D1 to D9, and includes, for example, two cameras 4a, 4b, and a projector 4c. The projector 4c projects a specified pattern onto the region where the plurality of workpieces D1 to D9 exist, and the two cameras 4a, 4b photograph the existence regions of the plurality of workpieces onto which the specified pattern has been projected by the projector 4c, and measure the three-dimensional shapes of the workpieces D1 to D9.

[0025] In this way, the acquisition unit 4 can measure the three-dimensional shapes of the existence regions of the plurality of workpieces D1 to D9 and acquire three-dimensional measurement data. And the acquisition unit 4 can acquire, for example, a two-dimensional image obtained by photographing the existence regions of the plurality of workpieces D1 to D9 by using an image captured by one of the two cameras 4a, 4b.

[0026] Here, the acquisition unit 4 is not limited to the above structure. For example, three-dimensional measurement data of the existence regions of a plurality of workpieces can also be acquired by stereo cameras 4a, 4b, and a two-dimensional image of the existence regions of the plurality of workpieces can be acquired by a two-dimensional camera 4c. And as the acquisition unit 4, as long as it can acquire a two-dimensional image and three-dimensional measurement data of the existence regions of the workpieces D1 to D9, various other structures can be applied. In addition, in Figure 1 this case, the two-dimensional image and three-dimensional measurement data are output from the acquisition unit 4 to the position correction device 3, but the position correction device 3 can also, for example, only receive the image data captured by each camera 4a to 4c of the acquisition unit 4 for processing, and generate the two-dimensional image and three-dimensional measurement data (three-dimensional point cloud data) internally.

[0027] The position correction device 3 performs position correction processing of the workpiece based on the two-dimensional image and three-dimensional measurement data from the acquisition unit 4, or based on the two-dimensional image and three-dimensional measurement data generated from the image data of each of the cameras 4a to 4c of the acquisition unit 4. In addition, for example, when the load of the calculation process is small, the position correction device 3 may not be set as a dedicated position correction device 3, and may be built into the robot control device 2, for example. On the contrary, for example, when the load of the calculation process such as machine learning is large (large amount of calculation and data), the position correction device 3 may also be composed of a dedicated workstation provided near the robot control device 2, or a host computer or general-purpose computer provided at a location far from the robot system 100. And, when a large amount of data is input into the machine learning model for learning, a general-purpose computer or processor can be used. If GPGPU (General-Purpose computing on Graphics Processing Units), a large-scale PC cluster, etc. are applied, the processing can be performed at a higher speed.

[0028] Next, a first embodiment and a second embodiment of the position correction program and the position correction device of the present embodiment will be described. First, generally speaking, the position correction device of the first embodiment uses three-dimensional measurement data (three-dimensional point cloud data) to perform correction calculation on the shape area of the workpiece (object) calculated from the two-dimensional image, and corrects the position and posture or external dimensions, etc. of the workpiece based on the corrected shape area. And, the position correction device of the second embodiment uses the image processing result of the two-dimensional image to perform correction calculation on the shape area of the workpiece calculated from the three-dimensional measurement data, and corrects the position and posture or external dimensions, etc. of the workpiece based on the corrected shape area.

[0029] Here, with reference to Figure 6 the above-mentioned "shape area of the workpiece" will be described. Figure 6 FIG. is a diagram for explaining the shape area of the workpiece in the first and second embodiments of the position correction device of the present embodiment. In this specification, as shown in the left diagram of Figure 6 the expression "shape area of the workpiece" is, for example, the area (the area of the workpiece D itself) indicated by reference numeral A1 that only includes the workpiece D and does not include the background area around the workpiece D in the two-dimensional image or three-dimensional image (image generated from three-dimensional measurement data) P3 output from the acquisition unit 4. That is, as Figure 6As shown in the right figure, for example, in a two-dimensional image (three-dimensional image) P3, it includes the workpiece D and the surrounding background area, rather than the area shown by the reference numeral A2 (prescribed shape area) which is not a predetermined prescribed shape. That is, the "shape area of the workpiece" in this specification is "area information reflecting the shape / outer shape of the workpiece", and the shape / outer shape of the workpiece can be calculated using this information.

[0030] Figure 2 is a flowchart for explaining an example of the processing in the first embodiment of the position correction program (position correction device) of the present embodiment. Figure 3 is a diagram for explaining an example of the identification process of the workpiece in the first embodiment of the position correction device of the present embodiment. As Figure 2 shown, when an example of the processing in the position correction program of the first embodiment starts (START), in step ST11, a two-dimensional image and three-dimensional measurement data of the existence areas of a plurality of workpieces are acquired. That is, as described with reference to Figure 1 the acquisition unit 4 acquires a two-dimensional image and three-dimensional measurement data (three-dimensional point cloud data) of the existence areas of a plurality of workpieces (for example, a plurality of corrugated cardboard boxes) D1 to D9 and outputs them to the position correction device 3.

[0031] Next, it proceeds to step ST12, and the workpiece is detected based on the two-dimensional image. That is, the position correction device 3 detects the workpiece based on the two-dimensional image from the acquisition unit 4. As an example, Figure 3 shows a case where in the two-dimensional image P1 obtained by photographing the existence areas of a plurality of workpieces, there is almost no difference in the shading and color tone (hue, brightness, and chroma) of the entire workpiece (object) B0. That is, in the two-dimensional image P1, for example, when the shading and color tone of the workpiece B0 are substantially the same, the position correction device 3 detects the workpiece B0 as one workpiece (B1) and proceeds to step ST13.

[0032] In step ST13, the detection result of the workpiece is corrected based on the three-dimensional measurement data. Specifically, in the three-dimensional measurement data from the acquisition unit 4, for example, when the heights (three-dimensional shapes) of the workpieces B11, B12, and B13 are different respectively, and the workpiece B0 in the two-dimensional image P1 is not one workpiece B1 but three workpieces B11, B12, and B13, the detection result (for example, the detection quantity, detection position and posture, outer dimension, etc.) is corrected. That is, when the workpiece B0 is misrecognized as one workpiece B1 based on the two-dimensional image, the position correction device 3 compares the detection result with the three-dimensional measurement data, and corrects the detection quantity from one to three, such as the three workpieces B11, B12, and B13, and correctly corrects and outputs data such as the detection position and posture or outer dimension.

[0033] Here, in step ST12, the two-dimensional image used by the position correction device 3 in the detection of the workpiece, that is, the two-dimensional image of the existence regions of the plurality of workpieces obtained by the acquisition unit 4, can be, for example, either a black-and-white image (grayscale image) or a color image (RGB image). In addition, in step ST13, the three-dimensional measurement data used by the position correction device 3 in the correction of the workpiece detection result, that is, the three-dimensional measurement data of the existence regions of the plurality of workpieces obtained by the acquisition unit 4, only needs to be data capable of obtaining height direction information, for example.

[0034] Then, it proceeds to step ST14. After the motion plan of the robot is performed, it proceeds to step ST15 to control the robot 1 to pick up the workpiece. That is, in step ST14, when there are three workpieces B11, B12, and B13, the robot control device 2 performs the motion plan of the robot 1 according to the output data of the position correction device 3 such as the position and posture information of each workpiece, so as to pick up all the workpieces that can be detected / identified. And in step ST15, the robot control device 2 outputs a control command to the robot mechanism unit 10, and the robot mechanism unit 10 receives the control command to perform the picking-up action. For example, the robot 1 picks up the three workpieces B11, B12, and B13 in sequence. Then, an example of the processing in the first embodiment of the position correction program of the present embodiment is ended (END).

[0035] In the above example, for instance, although there are actually three workpieces B11, B12, and B13, without correction, the robot mechanism unit 10 may move to the wrong / offset detection position and attempt to pick up workpiece B1, which may cause failures such as being unable to pick up workpiece B1 or picking up workpiece B1 by contacting the lower right corner of the actual workpiece B11 and then losing balance during subsequent operations, resulting in the dropping of workpiece B11. In such a case, according to the position correction device 3 of the present first embodiment, by correcting the detection results such as the number of detected workpieces and the detection position and posture, failures in the picking-up action can be prevented.

[0036] In this way, the first embodiment of the position correction program (position correction device) of the present embodiment calculates at least the detection position of at least one workpiece based on the two-dimensional image obtained by photographing the existence regions of the plurality of workpieces, and corrects and outputs the detection position of at least one workpiece according to the three-dimensional measurement data obtained by measuring the existence regions of the plurality of workpieces. The position correction device 3 of the first embodiment, for example, uses the three-dimensional measurement data to perform a correction calculation on the shape region of the workpiece calculated based on the two-dimensional image, and performs a correction calculation on the position and posture or external dimensions, etc. of the workpiece according to the corrected shape region.

[0037] Here, in order to calculate the shape regions of multiple workpieces (e.g., multiple boxes, corrugated papers, etc.) that appear in the two-dimensional image from the acquisition unit 4, the position correction device 3 can perform the calculation using the learning results of image processing or machine learning (e.g., the learned complete model). For example, for the existence regions of multiple workpieces arranged in different multiple arrangement patterns, multiple two-dimensional images are captured using the stereo cameras (4a, 4b) of the acquisition unit 4, and three-dimensional measurement is performed simultaneously to obtain three-dimensional point cloud data (three-dimensional measurement data). In addition, the shape regions of each workpiece that appear on the captured multiple two-dimensional images are taught, and the teaching result and the image are generated as learning data. For such learning data, deep learning is performed using, for example, Fast R-CNN (Region Based Convolutional Neural Networks), Faster R-CNN, or Mask R-CNN, etc., to generate a learned complete model. And when the position, posture, arrangement pattern of the workpiece relative to the camera (acquisition unit 4), or the type, number, etc. of the workpiece are changed, the acquisition unit 4 newly captures a two-dimensional image, and the learned complete model is used to predict and calculate the shape regions of the multiple workpieces that appear in the captured image. It can also be performed by other devices: the process of generating the learning data of machine learning, and the process of performing machine learning to generate a learned complete model. In this case, the position correction device 3 receives the learned complete model data to calculate the shape region of the workpiece. In addition, in the case of using image processing, of course, various known image processing techniques (e.g., pattern matching, edge extraction, etc.) can be used.

[0038] Then, the position correction device 3 performs a comparison between the three-dimensional measurement data (e.g., three-dimensional point cloud data or three-dimensional image) and the two-dimensional image for the calculation results of the shape regions in the two-dimensional images of the multiple workpieces, and uses the three-dimensional position differences included in the three-dimensional measurement data to exclude the background regions, obstacle regions, or adjacent workpiece regions, etc. that are erroneously included in the calculation results. In addition, the position correction device 3 performs a comparison between the three-dimensional measurement data and the two-dimensional image, calculates features based on the three-dimensional measurement data, uses the calculated features to correct the shape regions of the workpieces in the two-dimensional image, and corrects and outputs the detection positions of the workpieces according to the corrected shape regions. Based on such corrected shape regions, for example, the centroid position of the shape region can be calculated as the detection position of the workpiece and output. In addition, the robot control device 2 controls the robot 1 (robot mechanism unit 10) according to the corrected detection position from the position correction device 3, so that the robot 1 performs the workpiece picking process. In addition, as a feature, the position correction device 3 calculates at least one of a groove, a gap, a step, a three-dimensional plane, and a three-dimensional curved surface. And the position correction device 3 corrects and calculates and outputs at least one of the posture, outer shape, and size of at least one workpiece and the detection position of at least one workpiece.

[0039] However, when the color and brightness of the workpiece shown in the two-dimensional image captured by the acquisition unit 4 (camera) are close to those of the surrounding background, various misidentifications or non-detections may occur as a result of workpiece recognition using machine learning or image processing. That is, for example, sometimes the background area around the workpiece is also misidentified as part of the workpiece, or a part of the workpiece is misidentified as the background area. In these cases, the shape area of the workpiece and the background area cannot be correctly distinguished, and a size larger or smaller than the actual size is misdetected as the workpiece size. Or, sometimes a part of the background area where there is no workpiece is misidentified as the shape area of the workpiece. In addition, a non-detection situation may occur where the shape area of the workpiece is misidentified as the background area and the workpiece in the background cannot be detected.

[0040] Therefore, the position correction device 3 of the first embodiment calculates the three-dimensional position difference between the two using the three-dimensional position information included in the three-dimensional measurement data (three-dimensional point cloud data) corresponding to the shape area and the background area of the workpiece on the two-dimensional image. When the three-dimensional position difference between the shape area and the background area of the workpiece is large, for example, exceeds a predetermined threshold, it is determined that these two areas are different (not areas within the same workpiece), and the shape area of the workpiece and the background area can be correctly distinguished, preventing misidentifications (false detections) between the shape area and the background area of the workpiece, non-detection of the workpiece, and false detection of the background.

[0041] Moreover, when multiple workpieces (such as boxes, corrugated papers, etc.) shown in the two-dimensional image captured by the acquisition unit 4 are in close contact, it is difficult to identify the gaps (gaps, grooves, steps, etc.) between the workpieces based on the image, and multiple workpieces may be misidentified as one workpiece. In particular, the shading information of a grayscale two-dimensional image without RGB information is not as rich as that of an RGB image, so it is difficult to identify features such as gaps, grooves, and steps based on the limited information. The same problem also exists in a two-dimensional image with low resolution, for example.

[0042] Therefore, the position correction device 3 of the first embodiment calculates features such as gaps, grooves, and steps using the three-dimensional measurement data acquired by the acquisition unit 4 to identify the gaps between the workpieces, thereby preventing multiple closely contacted workpieces from being misidentified as one.

[0043] In addition, when the workpiece shown in the two-dimensional image captured by the acquisition unit 4 is a workpiece with uneven color distribution, one workpiece may be misrecognized as two or more workpieces. Therefore, the position correction device 3 of the first embodiment calculates features such as a three-dimensional plane or a three-dimensional curved surface using the three-dimensional measurement data acquired by the acquisition unit 4. For example, it is determined that a plurality of regions with different colors are actually regions on the same plane or curved surface, thereby preventing one workpiece from being misrecognized as two or more workpieces.

[0044] As described above, according to the first embodiment of the present invention, for example, even when the color and brightness of the workpiece (object) shown in the two-dimensional image are close to those of the surrounding background, the three-dimensional position difference between the workpiece and the background based on the three-dimensional measurement data can be used to correctly distinguish the shape region of the workpiece and the background region, preventing misrecognition between the workpiece and the background or non-detection of the workpiece. In addition, according to the first embodiment of the present invention, by using the three-dimensional measurement data to identify the gap (such as a gap, groove, step, etc.) between two workpieces in close contact in the two-dimensional image, it is possible to prevent two workpieces from being misrecognized as one. And, according to the first embodiment of the present invention, for example, even when the color distribution of the workpiece shown in the two-dimensional image is uneven, it is confirmed in the three-dimensional measurement data that these regions are regions on the same plane or curved surface, thereby also preventing one workpiece from being misrecognized as two or more workpieces.

[0045] Figure 4 It is a flowchart for explaining an example of the processing in the second embodiment of the position correction program (position correction device) of the present embodiment. Figure 5 It is a diagram for explaining an example of the recognition processing of the workpiece in the second embodiment of the position correction device of the present embodiment. As Figure 4 shown, when an example of the processing in the position correction program of the second embodiment starts (START), in step ST21, three-dimensional measurement data and two-dimensional images of the existence regions of a plurality of workpieces are acquired. That is, as described with reference to Figure 1 the acquisition unit 4 acquires two-dimensional images and three-dimensional measurement data of the existence regions of a plurality of workpieces D1 to D9 and outputs them to the position correction device 3.

[0046] Next, it proceeds to step ST22, and the workpiece is detected based on the three-dimensional measurement data. That is, the position correction device 3 detects the workpiece based on the three-dimensional measurement data from the acquisition unit 4. As an example, Figure 5This indicates a situation where, in the three-dimensional measurement data P2 obtained by photographing the presence areas of multiple workpieces (for example, a three-dimensional image generated based on three-dimensional point cloud data), there is almost no difference in the height (three-dimensional shape) of the entire workpiece (object) C0. That is, in the three-dimensional measurement data P2, when the heights of the workpieces C0 are approximately the same, the position correction device 3 detects the workpiece C0 as one workpiece (C1) and proceeds to step ST23.

[0047] In step ST23, the detection result of the workpiece is corrected based on the two-dimensional image. Specifically, in the two-dimensional image from the acquisition unit 4, for example, when there are significant differences in the shading and color tone (hue, brightness, and chroma) of the workpieces C11, C12, and C13, if the workpiece C0 in the three-dimensional measurement data P2 is not one workpiece C1 but three workpieces C11, C12, and C13, the detection result (such as the number of detections, detection position and pose, external dimensions, etc.) is corrected. That is, the position correction device 3 compares the detection result of regarding the workpiece C0 as one workpiece C1 based on the three-dimensional measurement data with the two-dimensional image. As a result, the number of detections is corrected from one to three, such as for the three workpieces C11, C12, and C13, and data such as the detection position and pose or external dimensions is correctly corrected and output.

[0048] Then, it proceeds to step ST24. After making a motion plan for the robot, it proceeds to step ST25 to control the robot 1 to pick up the workpiece. That is, in step ST24, when there are 3 workpieces C11, C12, and C13, the robot control device 2 makes a motion plan for the robot 1 based on the output data of the position correction device 3 such as the position and pose information of each workpiece, in order to pick up all the workpieces that can be detected / identified. And in step ST25, the robot control device 2 outputs a control command to the robot mechanism unit 10, for example. The robot mechanism unit 10 receives the control command and performs the pick-up action, so that the robot 1 sequentially picks up the three workpieces C11, C12, and C13. Then, an example of the processing in the second embodiment of the position correction program of this embodiment ends (END).

[0049] In the above example, for instance, although there are actually 3 workpieces C11, C12, and C13, without correction, the robot mechanism unit 10 may move to the wrong / offset detection position and attempt to pick up the workpiece C1, which may cause failures such as being unable to pick up the workpiece C1 and failing, or picking up the workpiece C12 by contacting a position that is substantially offset from the center of gravity of the workpiece C12, and then losing balance during subsequent operations and causing the workpiece C12 to fall. In such a case, according to the position correction device 3 of this second embodiment, by correcting the detection result such as the number of detections and the detection position and pose of the workpiece, failures in the pick-up action can be prevented.

[0050] Thus, the second embodiment of the position correction program (position correction device) of the present embodiment uses the image processing result of a two-dimensional image to perform a correction calculation on the shape area of the workpiece calculated from the three-dimensional measurement data, and corrects the position and orientation or external dimensions, etc. of the workpiece based on the corrected shape area. The position correction device 3 of the second embodiment, for example, calculates the detection position of the workpiece from the three-dimensional measurement data, and corrects the detection position of the workpiece based on the processing result of the two-dimensional image. That is, the position correction device 3 calculates the detection position of at least one workpiece from the three-dimensional measurement data obtained by measuring the presence areas of multiple workpieces, and corrects and outputs the detection position of at least one workpiece based on the two-dimensional image obtained by photographing the presence areas of multiple workpieces.

[0051] In addition, the position correction device 3 of this second embodiment calculates the shape area of the workpiece in the three-dimensional measurement data, and corrects and outputs the detection position of at least one workpiece based on the calculated shape area of the workpiece. Moreover, the position correction device 3 can be configured to calculate the shape area of the workpiece in the three-dimensional measurement data using the learning result of image processing or machine learning in the same manner as the first embodiment described above. Additionally, the position correction device 3 can also perform a matching process, etc. with the three-dimensional CAD (Computer Aided Design) model of the workpiece on the three-dimensional measurement data (for example, three-dimensional point cloud data) to calculate the shape area of the workpiece in the three-dimensional measurement data. Furthermore, in the case of using image processing, as described above, various known image processing techniques (for example, pattern matching, edge extraction, etc.) can be utilized.

[0052] Moreover, the position correction device 3 of this second embodiment can perform a comparison between the two-dimensional image and the three-dimensional measurement data, correct the shape area of the workpiece in the three-dimensional measurement data using the difference in pixel values included in the two-dimensional image, and correct and output the detection position of the workpiece based on the corrected shape area. Additionally, the position correction device 3 can also perform a comparison between the two-dimensional image and the three-dimensional measurement data, calculate features based on the two-dimensional image, use the calculated features to correct the shape area of the workpiece in the three-dimensional measurement data, and correct and output the detection position of the workpiece based on the corrected shape area.

[0053] Moreover, the position correction device 3 of this second embodiment can calculate at least one of the patterns of edges, grooves, gaps, steps, circles, planes, curved surfaces, and feature points as features. Additionally, the position correction device 3 can also calculate at least one of the postures, external shapes, and dimensions of at least one workpiece together with the detection position of at least one workpiece. And the position correction device 3 can also correct and output at least one of the postures, external shapes, and dimensions of at least one workpiece together with the detection position of at least one workpiece.

[0054] As described above, according to this second embodiment, for example, in the case of using an inexpensive three-dimensional measuring device, even when its measurement accuracy is low and the quality of the obtained three-dimensional measurement data is poor, by using the image processing result of the two-dimensional image captured by the inexpensive camera to correct the detection position and posture, etc., the detection accuracy of the position and posture of the workpiece can be improved. Thus, even without using a three-dimensional measuring device with high measurement accuracy and high price, it is possible to obtain detection results of high accuracy comparable to that, and therefore, the cost of introducing equipment can be reduced. In addition, according to this second embodiment, the two-dimensional image and the three-dimensional measurement data in the same area (shooting range: the area where multiple workpieces exist) are obtained for correction calculation. Therefore, for example, in the shooting method where the shooting range of the second image (three-dimensional image including three-dimensional point cloud data) is definitely smaller than the shooting range of the first image (two-dimensional camera image) as listed in [Prior Art Documents] [Patent Document 1], it is also possible to prevent the situation where the three-dimensional measurement data cannot be used in the calculation of the detection position of the workpiece in a part of the area due to the three-dimensional measurement data of the workpiece in that area not being obtained, resulting in a decrease in position accuracy.

[0055] Thus, the first and second embodiments of the position correction program (position correction device) of this embodiment capture two-dimensional images of the area where multiple workpieces exist, and perform three-dimensional measurement on the same area to obtain three-dimensional measurement data (for example, three-dimensional point cloud data). And by comparing the three-dimensional measurement data with the two-dimensional image, the detection (recognition) accuracy of the position, posture, or external dimensions, etc. of the workpiece can be improved. In addition, the first and second embodiments of the position correction program of the above-described embodiment of this invention can be executed by the arithmetic processing device in the robot control device 2 instead of the dedicated position correction device 3, for example, when the load of the calculation process is small. In this case, the position correction device 3 is assembled inside the robot control device 2. On the contrary, for example, when the load of the calculation process such as performing machine learning is large, the position correction device 3 can also be composed of a dedicated workstation provided near the robot control device 2, or a host computer or a general-purpose computer, etc. provided at a place far from the robot system 100. In addition, the position correction device 3 may not perform machine learning, but only accept the learning result (for example, learned complete model data) obtained by performing machine learning using other devices or services, such as a cloud service that can utilize a GPU device, for calculation processing.

[0056] The position correction program of the above-described embodiment can be provided by being recorded on a computer-readable non-transitory storage medium or a non-volatile semiconductor memory, and can also be provided via a wired or wireless manner. Here, as the computer-readable non-transitory storage medium, for example, optical discs such as CD-ROM (Compact Disc ReadOnly Memory) and DVD-ROM, or hard disk devices, etc. are considered. In addition, as the non-volatile semiconductor memory, PROM (Programmable Read Only Memory), flash memory, etc. are considered. And, as the distribution from the server device, it is considered to be provided via a wired or wireless LAN (Local Area Network), or a WAN such as the Internet.

[0057] As described in detail above, according to the position correction device, robot system, and position correction program of the present embodiment, it is possible to prevent the occurrence of false detection or non-detection problems and improve the recognition / detection accuracy of workpieces (objects).

[0058] The present disclosure has been described in detail, but the present disclosure is not limited to the above-described respective embodiments. These embodiments can be subjected to various additions, replacements, changes, partial deletions, etc. within the scope not departing from the spirit of the present disclosure, or within the scope not departing from the spirit of the present disclosure derived from the content described in the claims and its equivalents. In addition, these embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each action and the order of each process are shown as an example, but are not limited thereto. In addition, the same applies to the case where numerical values or mathematical formulas are used in the description of the above-described embodiments.

[0059] Regarding the above-described embodiments and modification examples, the following supplementary notes are also disclosed.

[0060] [Supplementary Note 1]

[0061] A position correction device (3), wherein,

[0062] Based on a two-dimensional image (P1) obtained by photographing the existence regions of a plurality of workpieces (D1 to D9, B0, D), at least the detection positions of at least one of the workpieces (B0, D) are calculated, and based on three-dimensional measurement data (P2) obtained by measuring the existence regions of the plurality of workpieces (D1 to D9, B0, D), the detection positions of at least one of the workpieces (B0) are corrected (B1, B11 to B13) and output.

[0063] [Supplementary Note 2]

[0064] The position correction device according to Supplementary Note 1, wherein,

[0065] The position correction device (3) calculates the shape region (A1) of the workpiece (B0, D) in the two-dimensional image (P1), and calculates the detection positions (B1, B11 to B13) of at least one of the workpieces (B0) based on the calculated shape region (A1) of the workpiece (B0, D).

[0066] [Supplementary Note 3]

[0067] The position correction device according to Supplementary Note 2, wherein,

[0068] The position correction device (3) calculates the shape region (A1) of the workpiece (B0, D) in the two-dimensional image (P1) by using the learning result of machine learning.

[0069] [Supplementary Note 4]

[0070] The position correction device according to Supplementary Note 2 or 3, wherein,

[0071] The position correction device (3) performs image processing on the two-dimensional image (P1) to calculate the shape region (A1) of the workpiece (B0, D) in the two-dimensional image (P1).

[0072] [Supplementary Note 5]

[0073] The position correction device according to any one of Supplementary Notes 2 to 4, wherein,

[0074] The position correction device (3) performs comparison between the three-dimensional measurement data (P2) and the two-dimensional image (P1), corrects the shape region (A1) of the workpiece (B0, D) in the two-dimensional image (P1) by using the three-dimensional position difference included in the three-dimensional measurement data (P2), and corrects and outputs the detection positions (B1, B11 to B13) of the workpiece (B0) based on the corrected shape region.

[0075] [Supplementary Note 6]

[0076] The position correction device according to any one of Supplementary Notes 2 to 5, wherein,

[0077] The position correction device (3) performs comparison between the three-dimensional measurement data (P2) and the two-dimensional image (P1), calculates features based on the three-dimensional measurement data (P2), corrects the shape region of the workpiece (B0, D) in the two-dimensional image (P1) by using the calculated features, and corrects and outputs the detection positions of the workpiece (B0, D) based on the corrected shape region.

[0078] [Supplementary Note 7]

[0079] The position correction device according to Note 6, wherein,

[0080] The position correction device (3) calculates at least one of a groove, a gap, a step, a three-dimensional plane, and a three-dimensional curved surface as the feature.

[0081] [Note 8]

[0082] A position correction device (3), wherein,

[0083] Based on the three-dimensional measurement data (P2) obtained by measuring the presence regions of a plurality of workpieces (D1 to D9, C0, D), at least the detection positions of at least one of the workpieces (C0, D) are calculated, and based on the two-dimensional image (P1) obtained by photographing the presence regions of the plurality of workpieces (D1 to D9, C0, D), the detection positions of at least one of the workpieces (C0) are corrected (C1, C11 to C13) and output.

[0084] [Note 9]

[0085] The position correction device according to Note 8, wherein,

[0086] The position correction device (3) calculates the shape regions (A1) of the workpieces (C0, D) in the three-dimensional measurement data (P2), and based on the calculated shape regions (A1) of the workpieces (C0, D), calculates the detection positions (C1, C11 to C13) of at least one of the workpieces (C0, D).

[0087] [Note 10]

[0088] The position correction device according to Note 9, wherein,

[0089] The position correction device (3) uses the learning result of machine learning to calculate the shape regions (A1) of the workpieces (C0, D) in the three-dimensional measurement data (P2).

[0090] [Note 11]

[0091] The position correction device according to Note 9 or Note 10, wherein,

[0092] The position correction device (3) processes the three-dimensional measurement data (P2) to calculate the shape regions (A1) of the workpieces (C0, D) in the three-dimensional measurement data (P2).

[0093] [Note 12]

[0094] The position correction device according to any one of Notes 9 to 11, wherein,

[0095] The position correction device (3) compares the two-dimensional image (P1) with the three-dimensional measurement data (P2), corrects the shape region (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) by using the difference in pixel values included in the two-dimensional image (P1), and corrects and outputs the detection position of the workpiece (C0, D) based on the corrected shape region (A1).

[0096] [Supplementary Note 13]

[0097] According to the position correction device according to any one of Supplementary Notes 9 to 12, wherein

[0098] The position correction device (3) compares the two-dimensional image (P1) with the three-dimensional measurement data (P2), calculates features based on the two-dimensional image (P1), corrects the shape region (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) by using the calculated features, and corrects and outputs the detection position of the workpiece (C0, D) based on the corrected shape region (A1).

[0099] [Supplementary Note 14]

[0100] According to the position correction device according to Supplementary Note 13, wherein

[0101] The position correction device (3) calculates at least one of patterns of edges, grooves, gaps, steps, circles, planes, curved surfaces, and feature points as the feature.

[0102] [Supplementary Note 15]

[0103] According to the position correction device according to any one of Supplementary Notes 1 to 14, wherein

[0104] The position correction device (3) calculates at least one of the postures, outer shapes, and dimensions of at least one of the workpieces (B0, C0, D) together with the detection positions of at least one of the workpieces (B0, C0, D).

[0105] [Supplementary Note 16]

[0106] According to the position correction device according to any one of Supplementary Notes 1 to 15, wherein

[0107] The position correction device (3) corrects and outputs at least one of the postures, outer shapes, and dimensions of at least one of the workpieces (B0, C0, D) together with the detection positions of at least one of the workpieces (B0, C0, D).

[0108] [Supplementary Note 17]

[0109] A robot system (100), comprising:

[0110] A robot (1) that performs a prescribed operation on workpieces (B0, C0, D);

[0111] An acquisition unit (4) that acquires a two-dimensional image (P1) obtained by photographing the existence regions of a plurality of workpieces (D1 to D9, B0, C0, D), and acquires three-dimensional measurement data (P2) of the existence regions of the plurality of workpieces (D1 to D9, B0, C0, D);

[0112] A position correction device (3) that calculates at least the detection position of at least one of the workpieces (B0, C0, D) based on the two-dimensional image (P1) and the three-dimensional measurement data (P2) from the acquisition unit (4), and corrects and outputs the detection position of at least one of the workpieces (B0, C0, D); and

[0113] A robot control device (2) that receives the output of the position correction device (3) and outputs a control command to the robot (1) to control the robot (1),

[0114] The position correction device (3) is the position correction device described in any one of Appendices 1 to 16.

[0115] [Appendix 18]

[0116] The robot system according to Appendix 17, wherein

[0117] The robot (1) is controlled such that the workpieces (B11 to B13, C11 to C13) are picked up at the detection position output by the position correction device (3) according to the control command output from the robot control device (2).

[0118] [Appendix 19]

[0119] A position correction program, wherein

[0120] The arithmetic processing device is caused to execute the following processing:

[0121] Based on a two-dimensional image (P1) obtained by photographing the existence regions of a plurality of workpieces (D1 to D9, B0, D), at least calculate the detection position of at least one of the workpieces (B0, D); and

[0122] Based on three-dimensional measurement data (P2) obtained by measuring the existence regions of the plurality of workpieces (D1 to D9, B0, D), correct and output the detection position of at least one of the workpieces (B0).

[0123] [Appendix 20]

[0124] A position correction program, wherein

[0125] the arithmetic processing unit is made to execute the following processing:

[0126] Based on three-dimensional measurement data (P2) obtained by measuring the presence regions of a plurality of workpieces (D1 to D9, C0, D), at least calculate the detection positions of at least one of the workpieces (C0, D); and

[0127] Based on a two-dimensional image (P1) obtained by photographing the presence regions of the plurality of workpieces (D1 to D9, C0, D), correct (C1, C11 to C13) and output the detection positions of at least one of the workpieces (C0).

[0128] Symbol Explanation

[0129] 1 Robot

[0130] 2 Robot control device

[0131] 3 Position correction device

[0132] 4 Acquisition unit (measurement unit)

[0133] 10 Robot mechanism unit

[0134] 11 Arm

[0135] 12 End effector (hand)

[0136] 100 Robot system

[0137] A1 Shape region

[0138] A2 Region

[0139] B0, B1, B11 to B13; C0, C1, C11 to C13; D1 to D9 Objects (workpieces).

Claims

1. A position correction device, characterized in that based on a two-dimensional image obtained by photographing the presence areas of a plurality of workpieces, at least the detection position of at least one of the workpieces is calculated, and based on three-dimensional measurement data obtained by measuring the presence areas of the plurality of workpieces, the detection position of at least one of the workpieces is corrected and output.

2. The position correction device according to claim 1, characterized in that the position correction device calculates the shape area of the workpiece in the two-dimensional image, and based on the calculated shape area of the workpiece, calculates the detection position of at least one of the workpieces.

3. The position correction device according to claim 2, characterized in that the position correction device uses the learning result of machine learning to calculate the shape area of the workpiece in the two-dimensional image.

4. The position correction device according to claim 2 or 3, characterized in that the position correction device performs image processing on the two-dimensional image to calculate the shape area of the workpiece in the two-dimensional image.

5. The position correction device according to any one of claims 2 to 4, characterized in that the position correction device performs comparison between the three-dimensional measurement data and the two-dimensional image, uses the three-dimensional position difference included in the three-dimensional measurement data to correct the shape area of the workpiece in the two-dimensional image, and corrects and outputs the detection position of the workpiece based on the corrected shape area.

6. The position correction device according to any one of claims 2 to 5, characterized in that the position correction device performs comparison between the three-dimensional measurement data and the two-dimensional image, calculates features based on the three-dimensional measurement data, uses the calculated features to correct the shape area of the workpiece in the two-dimensional image, and corrects and outputs the detection position of the workpiece based on the corrected shape area.

7. The position correction device according to claim 6, characterized in that the position correction device calculates at least one of a groove, a gap, a step, a three-dimensional plane, and a three-dimensional curved surface as the feature.

8. A position correction device, characterized in that based on three-dimensional measurement data obtained by measuring the presence areas of a plurality of workpieces, at least the detection position of at least one of the workpieces is calculated, and based on a two-dimensional image obtained by photographing the presence areas of the plurality of workpieces, the detection position of at least one of the workpieces is corrected and output.

9. The position correction device according to claim 8, characterized in that the position correction device calculates the shape area of the workpiece in the three-dimensional measurement data, and based on the calculated shape area of the workpiece, calculates the detection position of at least one of the workpieces.

10. The position correction device according to claim 9, characterized in that the position correction device uses the learning result of machine learning to calculate the shape area of the workpiece in the three-dimensional measurement data.

11. The position correction device according to claim 9 or 10, characterized in that The position correction device processes the three-dimensional measurement data to calculate the shape area of the workpiece in the three-dimensional measurement data.

12. The position correction device according to any one of claims 9 to 11, characterized in that the position correction device performs comparison between the two-dimensional image and the three-dimensional measurement data, and corrects the shape area of the workpiece in the three-dimensional measurement data by using the difference in pixel values included in the two-dimensional image, and corrects and outputs the detection position of the workpiece according to the corrected shape area.

13. The position correction device according to any one of claims 9 to 12, characterized in that the position correction device performs comparison between the two-dimensional image and the three-dimensional measurement data, calculates features according to the two-dimensional image, corrects the shape area of the workpiece in the three-dimensional measurement data by using the calculated features, and corrects and outputs the detection position of the workpiece according to the corrected shape area.

14. The position correction device according to claim 13, characterized in that the position correction device calculates at least one of patterns of edges, grooves, gaps, steps, circles, planes, curved surfaces, and feature points as the feature.

15. The position correction device according to any one of claims 1 to 14, characterized in that the position correction device calculates at least one of the postures, outer shapes, and dimensions of at least one workpiece together with the detection position of at least one workpiece.

16. The position correction device according to any one of claims 1 to 15, characterized in that the position correction device corrects and outputs at least one of the postures, outer shapes, and dimensions of at least one workpiece together with the detection position of at least one workpiece.

17. A robot system, characterized in that, comprising: a robot that performs a predetermined operation on a workpiece; an acquisition unit that acquires a two-dimensional image obtained by photographing the presence areas of a plurality of workpieces and acquires three-dimensional measurement data of the presence areas of the plurality of workpieces; a position correction device that calculates at least the detection position of at least one workpiece according to the two-dimensional image and the three-dimensional measurement data from the acquisition unit, and corrects and outputs the detection position of at least one workpiece; and a robot control device that receives the output of the position correction device and outputs a control command to the robot to control the robot, wherein the position correction device is the position correction device according to any one of claims 1 to 16.

18. The robot system according to claim 17, characterized in that the robot is controlled to pick up the workpiece at the detection position output by the position correction device according to the control command output from the robot control device.

19. A position correction program, characterized in that causes an arithmetic processing device to execute the following processing: calculate at least the detection position of at least one workpiece according to a two-dimensional image obtained by photographing the presence areas of a plurality of workpieces; Based on three-dimensional measurement data obtained by measuring the presence regions of the plurality of workpieces, correct and output the detection positions of at least one of the workpieces.

20. A position correction program, characterized in that causes an arithmetic processing device to perform the following processing: Based on three-dimensional measurement data obtained by measuring the presence regions of a plurality of workpieces, calculate at least the detection positions of at least one of the workpieces; Based on a two-dimensional image obtained by photographing the presence regions of the plurality of workpieces, correct and output the detection positions of at least one of the workpieces.

Images