Position correction device, robot system, and position correction program

Abstract

The purpose of the present invention is to provide a position correction device with which the occurrence of erroneous detection and nondetection issues are prevented and workpiece recognition / detection accuracy can be improved. This position correction device: at least calculates a detection position of at least one workpiece on the basis of a two-dimensional image in which the regions-of-existence of a plurality of workpieces are captured; and corrects, and outputs, the detection position of the at least one workpiece on the basis of three-dimensional measurement data obtained by measuring the regions-of-existence of the plurality of workpieces.

Description

Position correction device, robot system, and position correction program

[0001] The present disclosure relates to a position correction device, a robot system, and a position correction program.

[0002] Conventionally, for example, object detection technology is known that recognizes an object (workpiece) in an image captured by a camera and acquires information such as the object's position, orientation, and outer size. Also known is a method that uses image processing technology (e.g., pattern matching) to extract features from an image and recognizes an object using the extracted features.

[0003] Furthermore, in recent years, methods have been put into practical use that use machine learning to extract features from images, recognize objects in the images, and estimate the object's position, orientation, and external size, etc. Also proposed is a method that uses a 3D measuring device to measure the 3D shape of an object, and then uses the 3D measurement data (3D point cloud data) of the measured object to obtain the object's position, orientation, etc.

[0004] 2. Description of the Related Art Conventionally, various object detection techniques have been proposed for recognizing an object in an image captured by a camera and acquiring information such as the position, orientation, and outer size of the object.

[0005] JP 2016-192135 A JP 2010-120141 A

[0006] As described above, various object detection techniques have been proposed to date for recognizing an object in an image captured by a camera and acquiring information about the object's position, orientation, etc. In addition, in recent years, techniques have been put into practical use for estimating the object's position, orientation, etc. based on features extracted from an image using machine learning, and techniques have also been proposed for acquiring the object's position, orientation, etc. using three-dimensional measurement data of the object obtained using a three-dimensional measuring device.

[0007] However, in actual applications, conventional object detection techniques that acquire information such as the position, orientation, and external size of an object can sometimes erroneously recognize a single object as multiple objects, or multiple objects as a single object. Furthermore, when an object is close to its surrounding background, there are problems such as the inability to detect the object (undetection) or the erroneous recognition of the background as an object, even though the object does not actually exist.

[0008] Therefore, there is a demand for a position correction device, a robot system, and a position correction program that can prevent the occurrence of these erroneous detection and non-detection problems and improve the accuracy of workpiece (object) recognition / detection.

[0009] According to one embodiment of the present disclosure, a position correction device is provided that calculates at least the detection position of at least one workpiece based on a two-dimensional image capturing the presence area of ​​multiple workpieces, and corrects and outputs the detection position of at least one workpiece based on three-dimensional measurement data measuring the presence area of ​​multiple workpieces.

[0010] FIG. 1 is a diagram schematically illustrating an example of a robot system according to this embodiment. FIG. 2 is a flowchart for explaining an example of processing in a first example of a position correction program according to this embodiment. FIG. 3 is a diagram for explaining an example of workpiece recognition processing in the first example of a position correction device according to this embodiment. FIG. 4 is a flowchart for explaining an example of processing in a second example of a position correction program according to this embodiment. FIG. 5 is a diagram for explaining an example of workpiece recognition processing in the second example of a position correction device according to this embodiment. FIG. 6 is a diagram for explaining the shape region of a workpiece in the first and second examples of a position correction device according to this embodiment.

[0011] Hereinafter, examples of a position correction device, a robot system, and a position correction program according to the present embodiment will be described in detail with reference to the accompanying drawings. In each drawing, identical or similar components are assigned identical or similar reference numerals. Furthermore, the embodiments described below do not limit the technical scope and meaning of the terms of the invention described in the claims.

[0012] Fig. 1 is a diagram schematically illustrating an example of a robot system according to this embodiment. As shown in Fig. 1, the robot system 100 includes a robot 1, a robot control device 2, a position correction device 3, and an acquisition unit (measurement unit) 4. The robot 1 includes a robot mechanism unit 10, an arm 11, and an end effector (hand unit) 12.

[0013] The robot 1 is configured as, for example, a multi-axis robot, and an end effector 12 is provided at the tip of an arm 11. In FIG. 1, the end effector 12 is illustrated as a suction device (suction hand), but it goes without saying that this can be changed to various other devices depending on the target object (workpiece) to which the robot system is applied, the work content, etc. The robot mechanism unit 10 causes the robot 1 to perform a predetermined operation based on control commands from the robot control device 2. That is, the robot control device 2 receives the output of the position correction device 3, and generates control commands for causing the robot 1 to perform a predetermined operation based on, for example, a program or teaching data stored in an internal storage device, and outputs the control commands to the robot mechanism unit 10. Furthermore, although the position correction device 3 is illustrated as being independent from the robot control device 2, the position correction device 3 may be configured to be incorporated into the robot control device 2.

[0014] The acquisition unit 4 is for acquiring two-dimensional images and three-dimensional measurement data (three-dimensional point cloud data) of the area where multiple workpieces (e.g., multiple cardboard boxes) D1 to D9 exist, and includes, for example, two cameras 4a and 4b and a projector 4c. The projector 4c projects a predetermined pattern onto the area where the multiple workpieces D1 to D9 exist, and the two cameras 4a and 4b capture images of the area where the multiple workpieces exist onto which the predetermined pattern is projected by the projector 4c, and measure the three-dimensional shapes of the workpieces D1 to D9.

[0015] In this way, the acquisition unit 4 can measure the three-dimensional shapes of the areas where the multiple workpieces D1 to D9 exist and acquire three-dimensional measurement data. Furthermore, the acquisition unit 4 can also acquire a two-dimensional image of the areas where the multiple workpieces D1 to D9 exist, for example, by using an image captured by one of the two cameras 4a and 4b.

[0016] Here, the acquisition unit 4 is not limited to the configuration described above, and may, for example, acquire 3D measurement data of the presence area of ​​multiple workpieces using stereo cameras 4a and 4b, and acquire 2D images of the presence area of ​​multiple workpieces using 2D camera 4c. Furthermore, various other configurations can be applied to the acquisition unit 4 as long as they can acquire 2D images and 3D measurement data of the presence area of ​​workpieces D1 to D9. While the acquisition unit 4 outputs 2D images and 3D measurement data to the position correction device 3 in FIG. 1, the position correction device 3 may, for example, receive and process only image data captured by each of the cameras 4a to 4c of the acquisition unit 4, and internally generate 2D images and 3D measurement data (3D point cloud data).

[0017] The position correction device 3 performs workpiece position correction processing based on the two-dimensional images and three-dimensional measurement data from the acquisition unit 4, or based on two-dimensional images and three-dimensional measurement data generated from image data from each of the cameras 4a to 4c from the acquisition unit 4. Note that, for example, when the computational load is low, the position correction device 3 can be built into the robot control device 2 rather than being provided as a dedicated position correction device 3. Conversely, when the computational load is high (the amount of calculations and data is large), such as when performing machine learning, the position correction device 3 can be configured as a dedicated workstation installed near the robot control device 2, or a host computer or general-purpose computer installed in a location remote from the robot system 100. Furthermore, when inputting large amounts of data into a machine learning model for learning, general-purpose computers and processors can be used, but applying general-purpose computing on graphics processing units (GPGPUs) or large-scale PC clusters enables faster processing.

[0018] Next, first and second examples of the position correction program and position correction device according to this embodiment will be described. First, in summary, the position correction device of this first example performs correction calculations for the shape region of a workpiece (object) calculated based on a two-dimensional image using three-dimensional measurement data (three-dimensional point cloud data), and corrects the position, orientation, outer size, etc. of the workpiece based on the corrected shape region. Furthermore, the position correction device of this second example performs correction calculations for the shape region of the workpiece calculated based on the three-dimensional measurement data using the results of image processing of the two-dimensional image, and corrects the position, orientation, outer size, etc. of the workpiece based on the corrected shape region.

[0019] Here, the above-mentioned "workpiece shape region" will be described with reference to FIG. 6 . FIG. 6 is a diagram for explaining the workpiece shape region in the first and second examples of the position correction device according to this embodiment. In this specification, the term "workpiece shape region" refers to, for example, the region indicated by reference symbol A1 (the region of the workpiece D itself) in the two-dimensional image or three-dimensional image (generated from three-dimensional measurement data) P3 output from the acquisition unit 4, which includes only the workpiece D and does not include the background region around the workpiece D, as shown in the left diagram of FIG. 6 . That is, as shown in the right diagram of FIG. 6 , for example, the region indicated by reference symbol A2 in the two-dimensional image (three-dimensional image) P3 includes the workpiece D and the background region around it, but is not the region of a predetermined shape indicated by reference symbol A2 (the region of a predetermined shape). In other words, the "workpiece shape region" in this specification refers to "region information reflecting the shape / outline of the workpiece," and this information can be used to calculate the shape / outline of the workpiece.

[0020] FIG. 2 is a flowchart illustrating an example of processing in a first embodiment of the position correction program (position correction device) according to this embodiment, and FIG. 3 is a diagram illustrating an example of workpiece recognition processing in the first embodiment of the position correction device according to this embodiment. As shown in FIG. 2, when an example of processing in the position correction program according to the first embodiment starts (START), in step ST11, two-dimensional images and three-dimensional measurement data of the areas where multiple workpieces exist are acquired. That is, as described with reference to FIG. 1, the acquisition unit 4 acquires two-dimensional images and three-dimensional measurement data (three-dimensional point cloud data) of the areas where multiple workpieces (e.g., multiple cardboard boxes) D1 to D9 exist, and outputs them to the position correction device 3.

[0021] Next, the process proceeds to step ST12, where the workpiece is detected based on the two-dimensional image. That is, the position correction device 3 detects the workpiece based on the two-dimensional image from the acquisition unit 4. As an example, FIG. 3 shows a case where there is almost no difference in the shading or color (hue, brightness, and saturation) of the entire workpiece (object) B0 in a two-dimensional image P1 obtained by capturing an area where multiple workpieces are present. That is, for example, if the shading or color of the workpiece B0 is almost the same in the two-dimensional image P1, the position correction device 3 detects the workpiece B0 as a single workpiece (B1) and proceeds to step ST13.

[0022] In step ST13, the workpiece detection results are corrected based on the three-dimensional measurement data. Specifically, if the three-dimensional measurement data from the acquisition unit 4 shows that the heights (three-dimensional shapes) of the workpieces B11, B12, and B13 are different, the detection results (e.g., the number of detections, the detected position and orientation, the external size, etc.) are corrected to show that the workpiece B0 in the two-dimensional image P1 is not a single workpiece B1 but three workpieces B11, B12, and B13. That is, the position correction device 3 corrects the erroneous detection result that shows that the workpiece B0 is a single workpiece B1 based on the two-dimensional image by comparing it with the three-dimensional measurement data, correcting the number of detections from one to three so that there are three workpieces B11, B12, and B13, and correctly correcting and outputting data such as the detected position and orientation and external size.

[0023] Here, in step ST12, the two-dimensional image used by the position correction device 3 to detect the workpiece, i.e., the two-dimensional image of the area where the multiple workpieces exist acquired by the acquisition unit 4, may be, for example, either a black and white image (grayscale image) or a color image (RGB image). Also, in step ST13, the three-dimensional measurement data used by the position correction device 3 to correct the detection result of the workpiece, i.e., the three-dimensional measurement data of the area where the multiple workpieces exist acquired by the acquisition unit 4, may be data from which information in the height direction can be acquired, for example.

[0024] The process then proceeds to step ST14, where the robot's motion is planned, and then to step ST15, where the robot 1 is controlled to pick up the workpiece. That is, in step ST14, the robot control device 2 plans the motion of the robot 1 so that three workpieces B11, B12, and B13 are present and all detected / recognized workpieces are picked up based on the output data of the position correction device 3, such as information on the position and orientation of each workpiece. Furthermore, in step ST15, the robot control device 2 outputs a control command to the robot mechanism unit 10, for example, so that the robot 1 picks up the three workpieces B11, B12, and B13 in order. The robot mechanism unit 10 receives the control command and performs the pick-up operation. Then, the process ends (END) for one example of processing in the first example of the position correction program according to this embodiment.

[0025] In the above example, for example, even though three workpieces B11, B12, and B13 are actually present, without correction, the robot mechanism 10 may move to an incorrect / shifted detection position and attempt to pick up workpiece B1, resulting in failure to lift up workpiece B1, or the suction pad may actually contact the lower right corner of workpiece B11 to lift it, causing workpiece B11 to lose balance during subsequent handling and drop. In such cases, the position correction device 3 according to the first embodiment can correct the detection results, such as the number of detected works and the detected position and orientation, to prevent failure in the pick-up operation.

[0026] In this way, the first example of the position correction program (position correction device) according to this embodiment calculates the detection position of at least one workpiece based on a two-dimensional image capturing the areas where multiple workpieces exist, and corrects and outputs the detection position of at least one workpiece based on three-dimensional measurement data capturing the areas where multiple workpieces exist. The position correction device 3 of the first example, for example, corrects and calculates the shape area of ​​the workpiece calculated based on the two-dimensional image using the three-dimensional measurement data, and corrects and calculates the position, orientation, outer size, etc. of the workpiece based on the corrected shape area.

[0027] Here, the position correction device 3 can use the learning results (e.g., a trained model) of image processing or machine learning to calculate the shape regions of each of multiple workpieces (e.g., multiple boxes, cardboard boxes, etc.) captured in the 2D images from the acquisition unit 4. For example, the stereo cameras (4a, 4b) of the acquisition unit 4 capture multiple 2D images of the presence regions of multiple workpieces arranged in multiple different arrangements, and simultaneously perform 3D measurements to acquire 3D point cloud data (3D measurement data). The shape regions of each of the workpieces captured in the captured 2D images are then taught, and the teaching results and images are generated as training data. Deep learning is performed on such training data using, for example, Fast R-CNN (Region Based Convolutional Neural Networks), Faster R-CNN, or Mask R-CNN to generate a trained model. Furthermore, if the position, posture, arrangement, or type or number of workpieces relative to the camera (acquisition unit 4) changes, the acquisition unit 4 captures a new two-dimensional image and predicts and calculates the shape area of ​​each of the multiple workpieces captured in the captured image using the trained model. The process of generating machine learning learning data and the process of executing machine learning to generate a trained model may be performed by a separate device. In that case, the position correction device 3 receives the trained model data and calculates the shape area of ​​the workpiece. It goes without saying that when image processing is used, various known image processing techniques (e.g., pattern matching, edge extraction, etc.) can be used.

[0028] The position correction device 3 then compares the calculated results of the shape regions in the 2D images of each of the multiple workpieces with the 3D measurement data (e.g., 3D point cloud data or 3D images) and the 2D images. Using the differences in 3D positions contained in the 3D measurement data, the position correction device 3 eliminates erroneously included background regions, obstacle regions, or regions of adjacent workpieces in the calculation results. The position correction device 3 also compares the 3D measurement data with the 2D images, calculates features based on the 3D measurement data, corrects the shape regions of the workpieces in the 2D images using the calculated features, and corrects and outputs the detected position of the workpieces based on the corrected shape regions. Based on the corrected shape regions, the position of the center of gravity of the shape regions may be calculated and output as the detected position of the workpieces. The robot control device 2 controls the robot 1 (robot mechanism unit 10) based on the corrected detected positions from the position correction device 3, causing the robot 1 to perform a workpiece removal process. The position correction device 3 also calculates at least one of grooves, gaps, steps, three-dimensional planes, and three-dimensional curved surfaces as features. Furthermore, the position correction device 3 corrects and calculates at least one of the posture, outer shape, and size of the at least one workpiece, and outputs the detected position of the at least one workpiece.

[0029] However, when the color or brightness of a workpiece captured in a two-dimensional image captured by the acquisition unit 4 (camera) is similar to that of the surrounding background, various misrecognitions or non-detection results may occur when workpiece recognition is performed using machine learning or image processing. For example, the background area around the workpiece may be mistakenly recognized as part of the workpiece, or a portion of the workpiece may be mistakenly recognized as the background area. In these cases, the shape area of ​​the workpiece and the background area cannot be correctly distinguished, resulting in the workpiece being mistakenly detected as being larger or smaller than its actual size. Alternatively, a portion of the background area where no workpiece is present may be mistakenly recognized as the shape area of ​​the workpiece. Furthermore, the shape area of ​​the workpiece may be mistakenly recognized as the background area, resulting in a non-detection case where the workpiece in the background cannot be detected.

[0030] Therefore, the position correction device 3 of the first embodiment uses the three-dimensional position information contained in the three-dimensional measurement data (three-dimensional point cloud data) corresponding to the workpiece shape region and background region on the two-dimensional image to calculate the difference in three-dimensional position between them. If the difference in three-dimensional position between the workpiece shape region and the background region is large, for example exceeding a predetermined threshold, it is determined that the two regions are separate (not regions within the same workpiece), and the workpiece shape region and the background region can be correctly distinguished, preventing erroneous recognition (false detection) of the workpiece shape region and the background region, as well as non-detection of the workpiece and erroneous detection of the background.

[0031] Furthermore, if multiple workpieces (e.g., boxes, cardboard boxes, etc.) appear in a two-dimensional image captured by the acquisition unit 4 and are closely spaced, it becomes difficult to recognize the gaps (gaps, grooves, steps, etc.) between the workpieces from the image, and there is a risk that the multiple workpieces will be mistakenly recognized as a single workpiece. In particular, grayscale two-dimensional images that do not contain RGB information do not have as much information about shading as RGB images, making it difficult to recognize features such as gaps, grooves, and steps from the limited information. This same problem exists, for example, in two-dimensional images with low resolution.

[0032] Therefore, the position correction device 3 of this first embodiment uses the three-dimensional measurement data acquired by the acquisition unit 4 to calculate features such as gaps, grooves, and steps to recognize the gaps between workpieces, thereby preventing multiple workpieces that are close together from being mistakenly recognized as one.

[0033] Furthermore, if the workpiece shown in the two-dimensional image captured by the acquisition unit 4 has an uneven color distribution, there is a risk that one workpiece will be mistakenly recognized as two or more workpieces. Therefore, the position correction device 3 of the first embodiment uses the three-dimensional measurement data acquired by the acquisition unit 4 to calculate the characteristics of a three-dimensional plane or three-dimensional curved surface, and for example, determines that multiple areas of different colors are actually areas on the same plane or curved surface, thereby preventing one workpiece from being mistakenly recognized as two or more workpieces.

[0034] As described above, according to the first embodiment, even if the color and brightness of a workpiece (object) and its surrounding background in a two-dimensional image are similar, the difference in the three-dimensional positions of the workpiece and the background in the three-dimensional measurement data can be used to accurately distinguish between the shape region of the workpiece and the background, preventing misrecognition of the workpiece and the background and non-detection of the workpiece. Furthermore, according to the first embodiment, by recognizing gaps (gaps, grooves, steps, etc.) between two closely spaced workpieces in a two-dimensional image using the three-dimensional measurement data, it is possible to prevent the misrecognition of two workpieces as one. Furthermore, according to the first embodiment, even if the color distribution of the workpieces in a two-dimensional image is uneven, it is possible to prevent the misrecognition of one workpiece as two or more workpieces by confirming that the areas in the three-dimensional measurement data are on the same flat or curved surface.

[0035] FIG. 4 is a flowchart illustrating an example of processing in a second embodiment of the position correction program (position correction device) according to this embodiment, and FIG. 5 is a diagram illustrating an example of workpiece recognition processing in the second embodiment of the position correction device according to this embodiment. As shown in FIG. 4, when an example of processing in the position correction program according to the second embodiment starts (START), in step ST21, 3D measurement data and 2D images of the areas where multiple workpieces exist are acquired. That is, as described with reference to FIG. 1, the acquisition unit 4 acquires 2D images and 3D measurement data of the areas where multiple workpieces D1 to D9 exist and outputs them to the position correction device 3.

[0036] Next, the process proceeds to step ST22, where the workpiece is detected based on the three-dimensional measurement data. That is, the position correction device 3 detects the workpiece based on the three-dimensional measurement data from the acquisition unit 4. As an example, FIG. 5 shows a case where there is almost no difference in the overall height (three-dimensional shape) of the workpiece (object) C0 in three-dimensional measurement data P2 (e.g., a three-dimensional image generated from three-dimensional point cloud data) obtained by capturing an image of the area where multiple workpieces are present. That is, if the heights of the workpieces C0 are approximately the same in the three-dimensional measurement data P2, the position correction device 3 detects the workpieces C0 as a single workpiece (C1) and proceeds to step ST23.

[0037] In step ST23, the workpiece detection results are corrected based on the two-dimensional image. Specifically, if the two-dimensional image from the acquisition unit 4 shows significant differences in the shades and colors (hue, brightness, and thickness) of the workpieces C11, C12, and C13, the detection results (e.g., the number of detected workpieces, the detected position and orientation, the external size, etc.) are corrected to indicate that the workpiece C0 in the three-dimensional measurement data P2 is not a single workpiece C1 but three workpieces C11, C12, and C13. That is, the position correction device 3 compares the detection results, which show that the workpiece C0 is a single workpiece C1 based on the three-dimensional measurement data, with the two-dimensional image, correcting the number of detected workpieces from one to three so that there are three workpieces C11, C12, and C13, and outputting the corrected data, such as the detected position and orientation and external size.

[0038] The process then proceeds to step ST24, where the robot's motion is planned, and then to step ST25, where the robot 1 is controlled to pick up the workpiece. That is, in step ST24, the robot control device 2 plans the motion of the robot 1 so that three workpieces C11, C12, and C13 are present and all detected / recognized workpieces are picked up based on the output data of the position correction device 3, such as information on the position and orientation of each workpiece. Furthermore, in step ST25, the robot control device 2 outputs a control command to the robot mechanism unit 10, for example, so that the robot 1 picks up the three workpieces C11, C12, and C13 in order. The robot mechanism unit 10 receives the control command and performs the pick-up operation. Then, the process ends (END) for one example of processing in the second example of the position correction program according to this embodiment.

[0039] In the above example, for example, even though three workpieces C11, C12, and C13 are actually present, without correction, the robot mechanism 10 may move to an incorrect / shifted detection position and attempt to pick up workpiece C1, resulting in failure to lift up workpiece C1, or the suction pad may come into contact with workpiece C12 at a position shifted from its actual center of gravity, causing it to lose balance during subsequent handling and drop the workpiece C12. In such cases, the position correction device 3 according to the second embodiment can correct the detection results, such as the number of detected works and the detected positions and orientations of the works, thereby preventing failures in the pick-up operation.

[0040] In this way, the second example of the position correction program (position correction device) according to this embodiment corrects and calculates the shape region of a workpiece calculated based on three-dimensional measurement data using the image processing results of two-dimensional images, and corrects the position, orientation, outer size, etc. of the workpiece based on the corrected shape region. The position correction device 3 of the second example, for example, calculates the detected position of a workpiece based on three-dimensional measurement data, and corrects the detected position of the workpiece based on the processing results of the two-dimensional image. In other words, the position correction device 3 at least calculates the detected position of at least one workpiece based on three-dimensional measurement data obtained by measuring the presence regions of multiple workpieces, and at least corrects and outputs the detected position of at least one workpiece based on two-dimensional images obtained by capturing the presence regions of the multiple workpieces.

[0041] Furthermore, the position correction device 3 of this second embodiment calculates the shape region of the workpiece in the 3D measurement data and corrects and outputs at least one detected position of the workpiece based on the calculated shape region of the workpiece. Furthermore, as in the first embodiment described above, the position correction device 3 may be configured to calculate the shape region of the workpiece in the 3D measurement data using the learning results of image processing or machine learning. Furthermore, the position correction device 3 can also calculate the shape region of the workpiece in the 3D measurement data by performing, for example, matching processing on the 3D measurement data (e.g., 3D point cloud data) with a 3D CAD (Computer Aided Design) model of the workpiece. When using image processing, various known image processing techniques (e.g., pattern matching, edge extraction, etc.) can be used, as described above.

[0042] Furthermore, the position correction device 3 of this second embodiment may compare the two-dimensional image with the three-dimensional measurement data, use the difference in pixel values ​​contained in the two-dimensional image to correct the shape region of the workpiece in the three-dimensional measurement data, and correct and output the detected position of the workpiece based on the corrected shape region.The position correction device 3 may also compare the two-dimensional image with the three-dimensional measurement data, calculate features based on the two-dimensional image, use the calculated features to correct the shape region of the workpiece in the three-dimensional measurement data, and correct and output the detected position of the workpiece based on the corrected shape region.

[0043] The position correction device 3 of the second embodiment may calculate at least one of the following features: an edge, a groove, a gap, a step, a circle, a plane, a curved surface, and a feature point pattern. The position correction device 3 can also calculate at least one of the posture, shape, and size of at least one workpiece, along with the detected position of at least one workpiece. Furthermore, the position correction device 3 can correct and output at least one of the posture, shape, and size of at least one workpiece, along with the detected position of at least one workpiece.

[0044] As described above, according to the second embodiment, even if an inexpensive 3D measuring device has low measurement accuracy and the quality of the acquired 3D measurement data is poor, the detection accuracy of the workpiece position and orientation can be improved by correcting the detected position and orientation using the image processing results of a 2D image captured using an inexpensive camera. This allows for detection results of the position and orientation, etc., with similar accuracy to that of an expensive 3D measuring device with high measurement accuracy to be obtained, thereby reducing the cost of introducing equipment. Furthermore, according to the second embodiment, 2D images and 3D measurement data are acquired within the same area (imaging range: area where multiple workpieces exist) for correction calculations. Therefore, for example, as in Patent Document 1 listed in the "Prior Art Documents," where the imaging range of the second image (3D image including 3D point cloud data) is always smaller than the imaging range of the first image (2D camera image), this prevents a decrease in position accuracy due to the 3D measurement data not being acquired for some areas of the workpieces, making it impossible to use the 3D measurement data to calculate the detected positions of the workpieces within that area.

[0045] As described above, the first and second examples of the position correction program (position correction device) according to this embodiment capture two-dimensional images of the areas where multiple workpieces are present and perform three-dimensional measurements of the same areas to obtain three-dimensional measurement data (e.g., three-dimensional point cloud data). The three-dimensional measurement data is then compared with the two-dimensional images, thereby improving the accuracy of detecting (recognizing) the position, orientation, and external size of the workpieces. Note that the first and second examples of the position correction program according to this embodiment described above can be executed by an arithmetic processing unit in the robot control device 2, rather than a dedicated position correction device 3, when the computational load is low. In this case, the position correction device 3 is incorporated within the robot control device 2. Conversely, when the computational load is high, such as when performing machine learning, the position correction device 3 can be configured as a dedicated workstation located near the robot control device 2, or as a host computer or general-purpose computer located remotely from the robot system 100. In addition, the position correction device 3 may perform calculations by receiving only the learning results (e.g., trained model data) obtained by performing machine learning using another device or service, such as a cloud service that can use a GPU device, without performing machine learning.

[0046] The position correction program according to the present embodiment described above may be provided by being recorded on a computer-readable non-transitory recording medium or a non-volatile semiconductor memory, or may be provided via a wired or wireless connection. Examples of the computer-readable non-transitory recording medium include optical disks such as CD-ROMs (Compact Disc Read Only Memory) and DVD-ROMs, or hard disk drives. Examples of the non-volatile semiconductor memory include PROMs (Programmable Read Only Memory) and flash memory. Furthermore, distribution from a server device may include provision via a wired or wireless local area network (LAN) or a wide area network (WAN) such as the Internet.

[0047] As described above in detail, the position correction device, robot system, and position correction program according to this embodiment make it possible to prevent the occurrence of erroneous detection and non-detection problems, and improve the recognition / detection accuracy of the workpiece (object).

[0048] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0049] The following supplementary notes are further disclosed with respect to the above-described embodiment and modified examples. [Supplementary Note 1] A position correction device (3) that calculates at least a detection position of at least one workpiece (B0, D) based on a two-dimensional image (P1) capturing an image of an area where the workpieces (D1 to D9, B0, D) are present, and corrects (B1, B11 to B13) the detection position of at least one workpiece (B0) based on three-dimensional measurement data (P2) capturing the area where the workpieces (D1 to D9, B0, D) are present, and outputs the corrected position. [Supplementary Note 2] The position correction device (3) according to Supplementary Note 1 calculates a shape area (A1) of the workpiece (B0, D) in the two-dimensional image (P1), and calculates (B1, B11 to B13) the detection position of at least one workpiece (B0) based on the calculated shape area (A1) of the workpiece (B0, D). [Supplementary Note 3] The position correction device (3) calculates the shape region (A1) of the workpiece (B0, D) in the two-dimensional image (P1) by utilizing a learning result of machine learning. [Supplementary Note 4] The position correction device (3) according to Supplementary Note 2 or Supplementary Note 3, wherein the position correction device (3) calculates the shape region (A1) of the workpiece (B0, D) in the two-dimensional image (P1) by performing image processing on the two-dimensional image (P1). [Appendix 5] The position correction device (3) compares the three-dimensional measurement data (P2) with the two-dimensional image (P1), corrects the shape area (A1) of the workpiece (B0, D) in the two-dimensional image (P1) using a difference in three-dimensional position included in the three-dimensional measurement data (P2), and corrects (B1, B11 to B13) the detection position of the workpiece (B0) based on the corrected shape area, and outputs the corrected position. The position correction device according to any one of Appendices 2 to 4. [Supplementary Note 6] The position correction device (3) compares the three-dimensional measurement data (P2) with the two-dimensional image (P1), calculates features based on the three-dimensional measurement data (P2), corrects the shape region of the workpiece (B0, D) in the two-dimensional image (P1) using the calculated features, and corrects and outputs the detected position of the workpiece (B0, D) based on the corrected shape region. This is the position correction device described in any one of Supplementary Note 2 to Supplementary Note 5.[Supplementary Note 7] The position correction device (3) according to Supplementary Note 6, wherein the feature of the position correction device (3) is to calculate at least one of a groove, a gap, a step, a three-dimensional plane, and a three-dimensional curved surface. [Supplementary Note 8] The position correction device (3) calculates at least a detected position of at least one workpiece (C0, D) based on three-dimensional measurement data (P2) obtained by measuring the existence areas of a plurality of workpieces (D1 to D9, C0, D), and corrects (C1, C11 to C13) the detected position of at least one workpiece (C0) based on a two-dimensional image (P1) obtained by capturing the existence areas of the plurality of workpieces (D1 to D9, C0, D) and outputs the corrected position. [Supplementary Note 9] The position correction device according to Supplementary Note 8, wherein the position correction device (3) calculates a shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2), and calculates (C1, C11 to C13) the detection position of at least one of the workpieces (C0, D) based on the calculated shape area (A1) of the workpiece (C0, D). [Supplementary Note 10] The position correction device according to Supplementary Note 9, wherein the position correction device (3) calculates the shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) by utilizing a learning result of machine learning. [Supplementary Note 11] The position correction device according to Supplementary Note 9 or Supplementary Note 10, wherein the position correction device (3) processes the three-dimensional measurement data (P2) to calculate the shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2). [Appendix 12] The position correction device (3) compares the two-dimensional image (P1) with the three-dimensional measurement data (P2), corrects the shape region (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) using a difference in pixel values ​​contained in the two-dimensional image (P1), and corrects and outputs the detected position of the workpiece (C0, D) based on the corrected shape region (A1). This is the position correction device described in any one of Appendices 9 to 11.[Supplementary Note 13] The position correction device (3) compares the two-dimensional image (P1) with the three-dimensional measurement data (P2), calculates features based on the two-dimensional image (P1), corrects the shape region (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) using the calculated features, and corrects and outputs the detected position of the workpiece (C0, D) based on the corrected shape region (A1). [Supplementary Note 14] The position correction device (3) calculates at least one of an edge, a groove, a gap, a step, a circle, a plane, a curved surface, and a feature point pattern as the feature. [Supplementary Note 15] The position correction device according to any one of Supplementary Notes 1 to 14, wherein the position correction device (3) calculates at least one of the posture, outer shape, and size of the at least one workpiece (B0, C0, D) together with the detected position of the at least one workpiece (B0, C0, D). [Supplementary Note 16] The position correction device according to any one of Supplementary Notes 1 to 15, wherein the position correction device (3) corrects and outputs at least one of the posture, outer shape, and size of the at least one workpiece (B0, C0, D) together with the detected position of the at least one workpiece (B0, C0, D). [Supplementary Note 17] A robot (1) that performs a predetermined task on workpieces (B0, C0, D); an acquisition unit (4) that acquires a two-dimensional image (P1) that captures an area where a plurality of workpieces (D1 to D9, B0, C0, D) are present, and acquires three-dimensional measurement data (P2) of the area where the plurality of workpieces (D1 to D9, B0, C0, D) are present; a position correction device (3) that at least calculates a detected position of at least one of the workpieces (B0, C0, D) based on the two-dimensional image (P1) and the three-dimensional measurement data (P2) from the acquisition unit (4), and corrects and outputs the detected position of at least one of the workpieces (B0, C0, D); and a robot control device (2) that receives an output from the position correction device (3), and outputs a control command to the robot (1) to control the robot (1), A robot system (100) in which the position correction device (3) is the position correction device described in any one of Supplementary Notes 1 to 16.[Supplementary Note 18] The robot system according to Supplementary Note 17, wherein the robot (1) is controlled to pick up the workpieces (B11 to B13, C11 to C13) at the detection positions output by the position correction device (3) based on the control command output from the robot control device (2). [Supplementary Note 19] A position correction program that causes a calculation processing device to execute processes of: calculating at least a detection position of at least one of the workpieces (B0, D) based on a two-dimensional image (P1) capturing an image of an area where the plurality of workpieces (D1 to D9, B0, D) are present; and correcting and outputting the detection position of at least one of the workpieces (B0) based on three-dimensional measurement data (P2) measuring the area where the plurality of workpieces (D1 to D9, B0, D) are present. [Supplementary Note 20] A position correction program that causes a calculation processing device to execute a process of: calculating at least a detection position of at least one workpiece (C0, D) based on three-dimensional measurement data (P2) obtained by measuring the existence areas of a plurality of workpieces (D1 to D9, C0, D); and correcting (C1, C11 to C13) the detection position of at least one workpiece (C0) based on a two-dimensional image (P1) obtained by capturing the existence areas of the plurality of workpieces (D1 to D9, C0, D), and outputting the corrected position.

[0050] REFERENCE SIGNS LIST 1 Robot 2 Robot control device 3 Position correction device 4 Acquisition unit (measurement unit) 10 Robot mechanism unit 11 Arm 12 End effector (hand unit) 100 Robot system A1 Shape area A2 Area B0, B1, B11 to B13; C0, C1, C11 to C13; D1 to D9 Object (workpiece)

Claims

1. A position correction device that calculates at least the detection position of at least one workpiece based on a two-dimensional image capturing the presence areas of multiple workpieces, and corrects and outputs the detection position of at least one workpiece based on three-dimensional measurement data measuring the presence areas of the multiple workpieces.

2. The position correction device according to claim 1 , wherein the position correction device calculates a shape area of ​​the workpiece in the two-dimensional image, and calculates the detection position of at least one of the workpieces based on the calculated shape area of ​​the workpiece.

3. The position correction device according to claim 2 , wherein the position correction device calculates the shape region of the workpiece in the two-dimensional image by utilizing a learning result of machine learning.

4. The position correction device according to claim 2 or 3, wherein the position correction device performs image processing on the two-dimensional image to calculate the shape region of the workpiece in the two-dimensional image.

5. 4. The position correction device according to claim 2, wherein the position correction device compares the three-dimensional measurement data with the two-dimensional image, corrects the shape area of ​​the workpiece in the two-dimensional image using a difference in three-dimensional position contained in the three-dimensional measurement data, and corrects and outputs the detected position of the workpiece based on the corrected shape area.

6. 4. The position correction device according to claim 2, wherein the position correction device compares the three-dimensional measurement data with the two-dimensional image, calculates features based on the three-dimensional measurement data, corrects the shape region of the workpiece in the two-dimensional image using the calculated features, and corrects and outputs the detected position of the workpiece based on the corrected shape region.

7. 7. The position correction device according to claim 6, wherein the position correction device calculates at least one of a groove, a gap, a step, a three-dimensional plane, and a three-dimensional curved surface as the feature.

8. A position correction device that calculates at least the detection position of at least one workpiece based on three-dimensional measurement data obtained by measuring the presence areas of multiple workpieces, and corrects and outputs the detection position of at least one workpiece based on a two-dimensional image obtained by capturing the presence areas of the multiple workpieces.

9. 9. The position correction device according to claim 8, wherein the position correction device calculates a shape area of ​​the workpiece in the three-dimensional measurement data, and calculates the detected position of at least one of the workpieces based on the calculated shape area of ​​the workpiece.

10. The position correction device according to claim 9 , wherein the position correction device calculates the shape region of the workpiece in the three-dimensional measurement data by utilizing a learning result of machine learning.

11. The position correction device according to claim 9 or 10, wherein the position correction device processes the three-dimensional measurement data to calculate the shape region of the workpiece in the three-dimensional measurement data.

12. 11. The position correction device according to claim 9 or 10, wherein the position correction device compares the two-dimensional image with the three-dimensional measurement data, corrects the shape region of the workpiece in the three-dimensional measurement data using a difference in pixel values ​​contained in the two-dimensional image, and corrects and outputs the detected position of the workpiece based on the corrected shape region.

13. 11. The position correction device according to claim 9 or 10, wherein the position correction device compares the two-dimensional image with the three-dimensional measurement data, calculates features based on the two-dimensional image, corrects the shape region of the workpiece in the three-dimensional measurement data using the calculated features, and corrects and outputs the detected position of the workpiece based on the corrected shape region.

14. The position correction device according to claim 13 , wherein the position correction device calculates, as the feature, at least one of an edge, a groove, a gap, a step, a circle, a plane, a curved surface, and a pattern of feature points.

15. The position correction device according to any one of claims 1 to 3 and claims 8 to 10, wherein the position correction device calculates at least one of the posture, outer shape, and size of the at least one workpiece together with the detected position of the at least one workpiece.

16. The position correction device according to any one of claims 1 to 3 and claims 8 to 10, wherein the position correction device corrects and outputs at least one of the posture, shape, and size of at least one of the workpieces along with the detected position of at least one of the workpieces.

17. a robot that performs a predetermined task on a workpiece; an acquisition unit that acquires two-dimensional images of areas where a plurality of workpieces are present, and acquires three-dimensional measurement data of the areas where the plurality of workpieces are present; a position correction device that calculates at least a detection position of at least one of the workpieces based on the two-dimensional image and the three-dimensional measurement data from the acquisition unit, and corrects and outputs the detection position of the at least one of the workpieces; a robot control device that receives an output of the position correction device and outputs a control command to the robot to control the robot, The position correction device is a position correction device according to any one of claims 1 to 3 and claims 8 to 10. A robot system.

18. 18. The robot system according to claim 17, wherein the robot is controlled to pick up the workpiece at the detected position output by the position correction device based on the control command output from the robot control device.

19. The processing unit Calculating at least a detection position of at least one of the workpieces based on a two-dimensional image obtained by capturing an area where a plurality of workpieces are present; a position correction program that executes a process of correcting and outputting the detected position of at least one of the workpieces based on three-dimensional measurement data obtained by measuring the existence areas of the plurality of workpieces;

20. The processing unit Calculating at least a detection position of at least one of the workpieces based on three-dimensional measurement data obtained by measuring the presence areas of the plurality of workpieces; a position correction program that executes a process of correcting and outputting the detected position of at least one of the workpieces based on a two-dimensional image obtained by capturing an image of the presence area of ​​the plurality of workpieces;