POSITION CORRECTION DEVICE, ROBOT SYSTEM, AND POSITION CORRECTION PROGRAM

Abstract

The present invention aims to provide a position correction device that prevents the occurrence of false detections and missed detections and improves workpiece detection accuracy. This position correction device calculates at least one detection position of at least one workpiece based on a two-dimensional image in which the existence areas of a plurality of workpieces are captured; and corrects and outputs the detection position of the at least one workpiece based on three-dimensional measurement data obtained by measuring the existence areas of the plurality of workpieces.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a position correction device, a robot system and a position correction program. STATE OF THE ART

[0002] For example, an object recognition technique for identifying an object (workpiece) in an image captured by a camera and for capturing information about the object's position, location, size, outline, and the like is conventionally known. Furthermore, a technique for extracting a feature from an image using an image processing technique (e.g., pattern matching) and for recognizing an object using the extracted feature is also known.

[0003] Furthermore, in recent years a method for extracting a feature from an image using machine learning, for recognizing an object in the image, and for estimating the object's position, attitude, outline size, and the like has also been put into practice. A method for measuring an object's three-dimensional shape using a three-dimensional measuring device and for capturing the object's position, orientation, and the like using the measured three-dimensional measurement data (three-dimensional point group data) of the object has also been proposed.

[0004] Various proposals have been made so far for an object recognition technique to detect an object in an image taken by a camera and to capture information about the object's position, location, size of outline, and the like. CITATION LIST PATENT LITERATURE [PTL 1] Unexamined Japanese patent publication (Kokai) No. JP 2016-192135 A [PTL 2] Unexamined Japanese patent publication (Kokai) No. JP 2010-120141 A SUMMARY TECHNICAL PROBLEM

[0005] As described above, various conventional proposals have been made for an object recognition technique to detect an object in an image captured by a camera and to capture information about the object's position, orientation, and the like. In recent years, a technique for estimating an object's position, orientation, and the like based on a feature extracted from an image using machine learning has also been implemented, as has a technique for capturing an object's position, orientation, and the like using three-dimensional measurement data about the object obtained using a three-dimensional measuring instrument.

[0006] However, in the conventional object detection technique described above for capturing the position, location, outline size, and the like of an object, in actual application, for example, one object may be incorrectly recognized as a multitude of objects, or a multitude of objects may be incorrectly recognized as a single object. Furthermore, if an object and the background surrounding it are close together, there is the problem that the object cannot be detected (non-detection), and also the problem that the object is not actually present, but the background surrounding the object is incorrectly recognized as the object.

[0007] Therefore, it is desirable to provide a position correction device, a robot system, and a position correction program that can prevent the occurrence of the problem of false detection and non-detection and improve the detection / capture accuracy of a workpiece (object). SOLUTION TO THE PROBLEM

[0008] An embodiment according to the present disclosure provides a position correction device which, based on a two-dimensional image in which a presence area of ​​a plurality of workpieces is detected, calculates at least one detected position of at least one of the workpieces and corrects and outputs the detected position of at least one of the workpieces based on three-dimensional measurement data in which the presence area of ​​the plurality of workpieces is measured. BRIEF DESCRIPTION OF THE DRAWINGS [ Fig. 1] Fig. Figure 1 is a representation that schematically shows an example of a robot system according to the present embodiment. [ Fig. 2] Fig. Figure 2 is a flowchart describing an example of the processing in a first example of a position correction program according to the present embodiment. [ Fig. 3] Fig. Figure 3 is a representation to describe an example of the recognition processing of a workpiece in a first example of a position correction device according to the present embodiment. [ Fig. 4] Fig. Figure 4 is a flowchart describing an example of the processing in a second example of the position correction program according to the present embodiment. [ Fig. 5] Fig. Figure 5 is a representation describing an example of the recognition processing of a workpiece in a second example of the position correction device according to the present embodiment. [ Fig. 6] Fig. Figure 6 is a representation to describe a shape area of ​​a workpiece in the first and second examples of the position correction device according to the present embodiment. DESCRIPTION OF THE EXECUTION FORM

[0009] Examples of a position correction device, a robot system, and a position correction program according to the present embodiment are described in detail below with reference to the accompanying drawings. Identical or similar elements are identified in the drawings by the same or similar reference numerals. Furthermore, an embodiment described below does not limit the technical scope of the invention described in the claims or the meaning of any term.

[0010] Fig. Figure 1 is a schematic representation showing an example of a robot system according to the present embodiment. As in Fig. As shown in Figure 1, a robot system 100 comprises a robot 1, a robot controller 2, a position correction device 3, and a sensing unit (measuring unit) 4. The robot 1 comprises a robot mechanism unit 10, an arm 11, and an end effector (hand part) 12.

[0011] Robot 1, for example, is designed as a multi-axis robot, and the end effector 12 is attached to one end of arm 11. In Fig. The end effector 12 is an adsorption device (adsorption hand), but it goes without saying that the end effector 12 can be modified to various elements depending on the target object (workpiece) to which the robot system is applied, the work content, and the like. Note that the robot mechanism unit 10 serves to cause the robot 1 to perform a predetermined movement based on a control command from the robot controller 2. In other words, the robot controller 2 receives an output signal from the position correction device 3, generates a control command to cause the robot 1 to perform a predetermined movement, based, for example, on a program, training data, and the like that pre-stored in a memory device inside, and outputs the control command to the robot mechanism unit 10.Furthermore, the position correction device 3 is shown in such a way that it is independent of the robot control 2, but the position correction device 3 can be designed in such a way that it is integrated into the robot control 2.

[0012] The acquisition unit 4 is used to acquire a two-dimensional image and three-dimensional measurement data (three-dimensional point group data) across a presence area of ​​a large number of workpieces (e.g., a large number of boxes) D1 to D9 and comprises, for example, two cameras 4a and 4b and a projector 4c. The projector 4c projects a predefined pattern onto an area in which the multiple workpieces D1 to D9 are present, and the two cameras 4a and 4b capture the presence area of ​​the multiple workpieces onto which the predefined pattern is projected by the projector 4c and measure a three-dimensional shape of the workpieces D1 to D9.

[0013] In this way, the detection unit 4 can measure a three-dimensional shape of the presence area of ​​the multitude of workpieces D1 to D9 and acquire three-dimensional measurement data. Furthermore, the detection unit 4 can also, for example, acquire a two-dimensional image in which the presence area of ​​the multitude of workpieces D1 to D9 is captured using an image taken by one of the two cameras 4a and 4b.

[0014] The acquisition unit 4 is not limited to the configuration described above, but can, for example, use the stereo cameras 4a and 4b to acquire three-dimensional measurement data over the presence area of ​​a multitude of workpieces and the two-dimensional camera 4c to capture a two-dimensional image of the presence area of ​​the multitude of workpieces. As long as the acquisition unit 4 is capable of acquiring a two-dimensional image and three-dimensional measurement data over the presence area of ​​workpieces D1 to D9, other configurations for the acquisition unit 4 can also be used. Note that in Fig. 1. A two-dimensional image and three-dimensional measurement data are output from the acquisition unit 4 to the position correction device 3, but the position correction device 3 can, for example, only receive image data that was captured by each of the cameras 4a to 4c of the acquisition unit 4, process the image data and generate a two-dimensional image and three-dimensional measurement data (three-dimensional point group data) inside.

[0015] The position correction device 3 performs position correction processing of a workpiece based on a two-dimensional image and three-dimensional measurement data from the acquisition unit 4, or based on a two-dimensional image and three-dimensional measurement data generated from the image data of each of the cameras 4a to 4c of the acquisition unit 4. It should be noted that the position correction device 3 does not necessarily have to be a separate device if the computational effort is low, but can, for example, be integrated into the robot controller 2.Conversely, if the computational load for performing machine learning is high (a computational load and a data set are large), the position correction device 3 can also be formed from a host computer, a general-purpose computer, or the like, provided at a dedicated workstation near the robot controller 2 or at a location isolated from the robot system 100. If the learning is performed by inputting a large data set into a machine learning model, a general-purpose computer or processor can be used, but higher processing speeds can be achieved by applying general-purpose computing with graphics processing units (GPGPU), a large PC cluster, and the like.

[0016] The following describes a first and a second example of the position correction program and the position correction device according to the present embodiment. First, to give a brief summary, in the present first example, the position correction device uses three-dimensional measurement data (three-dimensional point group data) to correct and calculate a shape area of ​​a workpiece (object) calculated based on a two-dimensional image, and corrects the position, orientation, outline size, and the like of the workpiece based on the corrected shape area.Furthermore, in the present second example, the position correction device uses an image processing result of a two-dimensional image to correct and calculate a shape area of ​​a workpiece that was calculated based on three-dimensional measurement data, and corrects a position, a location, an outline size and the like of the workpiece based on the corrected shape area.

[0017] The following describes a “shape area of ​​a workpiece” as described above, with reference to Fig. 6 described. Fig. Figure 6 is a representation describing a shape area of ​​a workpiece in the first and second examples of the position correction device according to the present embodiment. In this description, the term "shape area of ​​a workpiece" includes only a workpiece D in a two-dimensional image or a three-dimensional image (generated from three-dimensional measurement data) P3 output by the acquisition unit 4, and denotes an area (a region of the workpiece D itself) that does not contain a background around the workpiece D and is indicated by a reference sign A1, as shown in a left-hand drawing in Figure 6. Fig. 6 shown. In other words, as for example in a right-hand drawing in Fig. As shown in Figure 6, the wording encompasses the workpiece D and the background area around the workpiece D in the two-dimensional image (three-dimensional image) P3 and does not specify an area (an area with a predetermined shape) that has a predetermined, predefined shape and is indicated by a reference sign A2. In other words, a "shape area of ​​a workpiece" in this specification is "area information that reflects a shape / outline of a workpiece," and the shape / outline of the workpiece can be calculated using this information.

[0018] Fig. Figure 2 is a flowchart describing an example of the processing in the first example of the position correction program (position correction device) according to the present embodiment, and Fig. Figure 3 is a representation describing an example of the workpiece recognition processing in the first example of the position correction device according to the present embodiment. As in Fig. As shown in Figure 2, when the processing in the position correction program begins in the first example, a two-dimensional image and three-dimensional measurement data are acquired over a presence area of ​​a large number of workpieces in stage ST11. In other words, as described in Figure 2, when the processing in the position correction program begins in the first example, a two-dimensional image and three-dimensional measurement data are acquired over a presence area of ​​a large number of workpieces in stage ST11. Fig. As described in 1, the acquisition unit 4 captures a two-dimensional image and three-dimensional measurement data (three-dimensional point group data) over a presence area of ​​the several workpieces (e.g. the several boxes) D1 to D9 and outputs the two-dimensional image and the three-dimensional measurement data to the position correction device 3.

[0019] The processing then continues with stage ST12, and the workpiece is detected based on the two-dimensional image. In other words, the position correction device 3 performs workpiece detection based on the two-dimensional image from the detection unit 4. As an example, Fig. 3. A case in which there is almost no difference in the shade and hue (hue, brightness, and saturation) of an entire workpiece (object) B0 in a two-dimensional image P1, in which a presence area of ​​a multitude of workpieces is captured. In other words, if, for example, the hue and tint of workpiece B0 in the two-dimensional image P1 are nearly identical, the position correction device 3 captures workpiece B0 as a single workpiece (B1), and processing continues with stage ST13.

[0020] In stage ST13, a workpiece measurement result is corrected based on the three-dimensional measurement data. Specifically, if workpieces B11, B12, and B13 have different heights (three-dimensional shapes) in the three-dimensional measurement data of the acquisition unit 4, a measurement result (e.g., a measured number, a measured position, a measured location, an outline size, and the like) is corrected so that workpiece B0 in the two-dimensional image P1 is not workpiece B1, but rather the three workpieces B11, B12, and B13.In other words, the position correction device 3 compares a false detection result based on the two-dimensional image, in which workpiece B0 is workpiece B1, with the three-dimensional measurement data, and thus corrects the detected number from one to three in order to define the three workpieces B11, B12 and B13, and correctly corrects and outputs data about a detected position, a detected location, an outline size and the like.

[0021] In stage ST12, the two-dimensional image used by the position correction device 3 to detect the workpiece, i.e., the two-dimensional image of the presence area of ​​the majority of workpieces captured by the detection unit 4, can be, for example, either a black and white image (grayscale image) or a color image (RGB image). Furthermore, in stage ST13, the three-dimensional measurement data used by the position correction device 3 to correct a detection result of the workpiece, i.e., the three-dimensional measurement data of the presence area of ​​the majority of workpieces captured by the detection unit 4, can be, for example, data from which information about a height direction can be obtained.

[0022] Furthermore, the process continues with stage ST14, where a motion plan for a robot is created. The processing then proceeds to ST15, where robot 1 is controlled and the workpiece is removed. In other words, in stage ST14, the robot controller 2 creates a motion plan for robot 1 such that all detectable workpieces are removed based on the output data from the position correction device 3. This data includes information about the presence of the three workpieces B11, B12, and B13, as well as the position, orientation, and other characteristics of each workpiece. Additionally, in stage ST15, for example, the robot controller 2 issues a control command to the robot mechanism unit 10, causing robot 1 to remove the three workpieces B11, B12, and B13 sequentially. The robot mechanism unit 10 receives the control command and executes the removal movement.Then one example of processing ends in the first example of the position correction program according to the present embodiment.

[0023] In the example described above, the robot mechanism unit 10 may, for instance, move into an incorrect / displaced detected position if no correction is made, even though the three workpieces B11, B12, and B13 are actually present. It may then attempt to remove workpiece B1, be unable to lift it, and fail. Alternatively, a fault may occur in which a substantially lower right corner of workpiece B11 comes into contact with and is lifted by an adsorption pad, resulting in a loss of balance during handling and the drop of workpiece B11. In such a case, the position correction device 3, as described in the first example, can prevent a removal movement failure and the like by correcting a detection result such as the detected number of workpieces, the detected position, and the detected orientation.

[0024] In the first example of the position correction program (position correction device) according to the present embodiment, at least one detected position of at least one of the workpieces is calculated based on a two-dimensional image in which a presence area of ​​the majority of workpieces is captured, and the detected position of at least one of the workpieces is corrected and output based on three-dimensional measurement data in which the presence area of ​​the majority of workpieces is measured. For example, in the first example, the position correction device 3 uses three-dimensional measurement data to correct and calculate a shape area of ​​a workpiece calculated based on a two-dimensional image, and corrects and calculates a position, orientation, outline size, and the like of the workpiece based on the corrected shape area.

[0025] In this context, the position correction device 3 can perform a calculation using a learning outcome (e.g., a learned model) from image processing or machine learning to calculate the shape area of ​​each of a plurality of workpieces (e.g., a plurality of boxes, cartons, and the like) captured in a two-dimensional image by the capture unit 4. For example, a plurality of two-dimensional images are captured using the stereo camera (4a, 4b) of the capture unit 4 with respect to the presence area of ​​a plurality of workpieces, each arranged in a plurality of different types of configurations, and three-dimensional point group data (three-dimensional measurement data) are acquired by simultaneously performing a three-dimensional measurement.Furthermore, a shape region of each workpiece, captured from the multitude of two-dimensional images taken, is trained, and a learning result and the image are generated as training data. Deep learning is performed on this training data using, for example, region-based convolutional neural networks (Fast R-CNN), Faster R-CNN, Mask R-CNN, or similar technologies, and a learned model is generated. Additionally, if a position, orientation, or arrangement of the workpiece relative to the camera (capture unit 4), the type and number of workpieces, or similar factors are changed, capture unit 4 re-captures a two-dimensional image, and a shape region of each of the multiple workpieces captured in the image is predicted and calculated using the learned model.The processing of the machine learning data and the creation of a learned model through machine learning can be performed by a separate device. In this case, the position correction device 3 receives learned model data and calculates a shape area of ​​a workpiece. It goes without saying that various well-known image processing techniques (e.g., pattern matching, edge extraction, and the like) can be used for image processing.

[0026] Then the position correction device 3 compares the three-dimensional measurement data (e.g., three-dimensional point group data or a three-dimensional image) and the two-dimensional image with a calculation result of the shape area in the two-dimensional image of each of the multitude of workpieces and excludes a background area, an obstacle area, an area of ​​an adjacent workpiece or the like that is erroneously included in the calculation result by using a difference in a three-dimensional position that is contained in the three-dimensional measurement data.Furthermore, the position correction device 3 compares the three-dimensional measurement data with the two-dimensional image, calculates a feature based on the three-dimensional measurement data, corrects the workpiece's shape area in the two-dimensional image using the calculated feature, and outputs a detected position of the workpiece based on the corrected shape area. Based on such a corrected shape area, for example, the position of the center of gravity of the shape area can be calculated and output as the detected position of the workpiece. Note that the robot controller 2 controls the robot 1 (robot mechanism unit 10) based on the corrected position detected by the position correction device 3 and instructs the robot 1 to remove the workpiece.Furthermore, the position correction device 3 calculates at least one groove, a gap, a step, a three-dimensional planar surface, or a three-dimensional curved surface as a feature. In addition, the position correction device 3 corrects, calculates, and outputs a detected position of at least one of the workpieces, as well as at least one position, outline, and size of at least one of the workpieces.

[0027] When the color and brightness of a workpiece, captured in a two-dimensional image by the capture unit 4 (camera), and a background surrounding the workpiece are closely related, various types of false detection and non-detection can occur as a result of job recognition using machine learning or image processing. In other words, for example, a background area around a workpiece might be incorrectly identified as part of the workpiece, or a part of the workpiece might be incorrectly identified as background. In these cases, the shape of the workpiece and the background cannot be correctly distinguished, and a size larger or smaller than the actual size is incorrectly detected as the size of the workpiece.Alternatively, a portion of a background area containing no workpiece may be incorrectly identified as the workpiece's shape area. Furthermore, a workpiece's shape area may be incorrectly identified as a background area, resulting in a missed detection scenario where a workpiece in the background cannot be detected.

[0028] Thus, in the present first example, the position correction device 3 uses three-dimensional position information contained in three-dimensional measurement data (three-dimensional point group data) corresponding to a shape area of ​​a workpiece and a background area on a two-dimensional image, and calculates a difference in a three-dimensional position between the shape area of ​​the workpiece and the background area.If the difference in the three-dimensional position between the workpiece's form area and the background area is large, for example, if the difference exceeds a predetermined threshold, it is determined that the two areas are different (not areas in the same workpiece), and the workpiece's form area and the background area can be correctly distinguished, and the occurrence of false detection (incorrect capture) between the workpiece's form area and the background area, failure to capture the workpiece, incorrect capture of the background, and the like can be prevented.

[0029] When multiple workpieces (e.g., boxes, cartons, and the like) visible in a two-dimensional image captured by the capture unit 4 are in close contact with each other, it is also difficult to detect any gaps (such as a crack, groove, or step) between the workpieces in the image, and the multiple workpieces may be mistakenly identified as a single workpiece. In particular, the color information in a two-dimensional grayscale image, which lacks RGB information, is not very extensive compared to an RGB image, and therefore it is difficult to identify a feature such as a crack, groove, or step based on the limited information. A similar problem exists, for example, with low-resolution two-dimensional images.

[0030] In the present first example, the position correction device 3 uses three-dimensional measurement data acquired by the detection unit 4, calculates a feature such as a gap, a groove and a step and detects a space between workpieces, thereby preventing several workpieces in close contact from being mistakenly recognized as one workpiece.

[0031] If a workpiece depicted in a two-dimensional image captured by the sensing unit 4 has a non-uniform color distribution, it may be incorrectly identified as two or more workpieces. Therefore, in this first example, the position correction device 3 uses three-dimensional measurement data acquired by the sensing unit 4, calculates a feature such as a three-dimensional planar surface or a three-dimensional curved surface, and determines that, for example, a multitude of areas with different colors are in fact areas on the same planar surface or the same curved surface. This prevents a workpiece from being incorrectly identified as two or more workpieces.

[0032] As described above, according to the first example, even if the color and brightness of a workpiece (object) captured in a two-dimensional image and a background around the workpiece are very similar, a shape area of ​​the workpiece and a background area can be correctly distinguished by using a difference in a three-dimensional position between the workpiece and the background through three-dimensional measurement data. This prevents the occurrence of false detection between the workpiece and the background, failure to detect the workpiece, and the like. According to the first example, a gap (such as a crack, groove, or step) between two workpieces in close contact on a two-dimensional image is detected using three-dimensional measurement data, thus preventing two workpieces from being incorrectly identified as one.Furthermore, according to the first example presented, even if a workpiece captured in a two-dimensional image has an uneven color distribution, three-dimensional measurement data confirms that the areas are areas on the same flat surface or the same curved surface, thus preventing a workpiece from being mistakenly recognized as two workpieces.

[0033] Fig. Figure 4 is a flowchart describing an example of the processing in the second example of the position correction program (position correction device) according to the present embodiment, and Fig. Figure 5 is a representation describing an example of the workpiece recognition processing in the second example of the position correction device according to the present embodiment. As in Fig. As shown in Figure 4, when one example of processing in the position correction program begins in the second example, three-dimensional measurement data and a two-dimensional image of a presence area of ​​a multitude of workpieces are acquired in stage ST21. In other words, as referenced in Fig. As described in 1, the acquisition unit 4 captures a two-dimensional image and three-dimensional measurement data over a presence area of ​​the majority of workpieces D1 to D9 and outputs the two-dimensional image and the three-dimensional measurement data to the position correction device 3.

[0034] The processing then proceeds to stage ST22, and the workpiece is detected based on the three-dimensional measurement data. In other words, the position correction device 3 performs the detection of the workpiece based on the three-dimensional measurement data from the detection unit 4. As an example, the following illustrates... Fig.5. A case in which there is almost no difference in the height (three-dimensional shape) of an entire workpiece (object) C0 in three-dimensional measurement data (for example, a three-dimensional image generated from three-dimensional point group data) P2, in which a presence area of ​​a multitude of workpieces is captured. In other words, if the height of workpiece C0 is nearly the same in the three-dimensional measurement data P2, the position correction device 3 captures workpiece C0 as a single workpiece (C1), and processing continues with stage ST23.

[0035] In stage ST23, a workpiece acquisition result is corrected based on the two-dimensional image. Specifically, if, for example, the shading and hue (a color, brightness, and saturation) of workpieces C11, C12, and C13 differ significantly in the two-dimensional image of acquisition unit 4, an acquisition result (for example, a captured number, a captured position, a captured location, an outline size, and the like) is corrected so that workpiece C0 in the three-dimensional measurement data P2 is not workpiece C1, and the three workpieces are C11, C12, and C13.In other words, the position correction device 3 compares the two-dimensional image with a detection result based on the three-dimensional measurement data, in which the workpiece C0 is a workpiece C1, and thus corrects the detected number from one to three in such a way that the three workpieces C11, C12 and C13 are determined, and correctly corrects and outputs data about a detected position, a detected location, an outline size and the like.

[0036] Furthermore, in stage ST24, a robot motion plan is created, and then in stage ST25, robot 1 is controlled and the workpiece is removed. In other words, in stage ST24, the robot controller 2 creates a motion plan for robot 1 such that all detectable workpieces are removed based on output data from the position correction device 3, which provides information about the presence of the three workpieces C11, C12, and C13, as well as the position, orientation, and other characteristics of each workpiece. Additionally, in stage ST25, for example, the robot controller 2 issues a control command to the robot mechanism unit 10, causing robot 1 to remove the three workpieces C11, C12, and C13 sequentially. The robot mechanism unit 10 receives the control command and executes a removal movement.Then one example of processing ends in the second example of the position correction program according to the present embodiment.

[0037] In the example described above, if no correction is made, the robot mechanism unit 10, even though the three workpieces C11, C12, and C13 are actually present, may move to an incorrect / displaced detected position, attempt to remove workpiece C1, be unable to lift it, and fail. Alternatively, a fault may occur where a significant portion, displaced from the center of gravity of workpiece C12, comes into contact with an adsorption pad and is lifted by it, resulting in a loss of balance during handling and the drop of workpiece C12. In such a case, the position correction device 3, as described in the second example, can prevent a removal movement failure and the like by correcting a detection result such as the detected number of workpieces, the detected position, and the detected orientation.

[0038] In this way, in the second example of the position correction program (position correction device) according to the present embodiment, a shape area of ​​a workpiece, calculated based on three-dimensional measurement data, is corrected and recalculated using an image processing result of a two-dimensional image, and the position, orientation, outline size, and the like of the workpiece are corrected based on the corrected shape area. For example, in the second example, the position correction device 3 calculates a detected position of a workpiece based on three-dimensional measurement data, and the detected position of the workpiece is corrected based on a processing result of a two-dimensional image.In other words, the position correction device 3 calculates at least one detected position of at least one of the workpieces based on three-dimensional measurement data in which a presence area of ​​the majority of workpieces is measured, and corrects and outputs at least the detected position of at least one of the workpieces based on a two-dimensional image in which the presence area of ​​the majority of workpieces is detected.

[0039] In the second example presented, the position correction device 3 calculates a shape area of ​​a workpiece in three-dimensional measurement data and corrects and outputs at least one detected position of at least one of the workpieces based on the acquired shape area. Furthermore, the position correction device 3 can be configured similarly to the first example described above to calculate a shape area of ​​a workpiece in three-dimensional measurement data using a learning outcome from image processing or machine learning. The position correction device 3 can also calculate a shape area of ​​a workpiece in three-dimensional measurement data (three-dimensional point group data) by, for example, performing a comparison process with a three-dimensional computer-aided design (CAD) model of the workpiece on the three-dimensional measurement data.Note that, as described above, when image processing is used, various well-known image processing techniques (e.g., pattern matching, edge extraction, and the like) can be employed.

[0040] Furthermore, in the second example presented, the position correction device 3 can compare a two-dimensional image with three-dimensional measurement data, correct a shape area of ​​a workpiece in the three-dimensional measurement data using a difference in a pixel value contained in the two-dimensional image, and correct and output a detected position of the workpiece based on the corrected shape area. Additionally, the position correction device 3 can compare a two-dimensional image with three-dimensional measurement data, calculate a feature based on the two-dimensional image, correct a shape area of ​​a workpiece in the three-dimensional measurement data using the calculated feature, and also correct and output a detected position of the workpiece based on the corrected shape area.

[0041] Then, in the second example presented, the position correction device 3 can calculate at least one edge, groove, gap, step, circle, flat surface, curved surface, or pattern of feature points as a feature. The position correction device 3 can also calculate a detected position of at least one workpiece, as well as at least one location, outline, and size of at least one workpiece. Furthermore, the position correction device 3 can also correct and output a detected location, outline, and size of at least one workpiece.

[0042] As described above, according to the second example presented, even if the measurement accuracy is low when using an inexpensive three-dimensional measuring instrument and the quality of the acquired three-dimensional measurement data is poor, the accuracy of determining the position, orientation, etc., of a workpiece can be improved by correcting the determined position, orientation, etc., using an image processing result from a two-dimensional image taken with an inexpensive camera. In this way, even without using a three-dimensional measuring instrument, which has high measurement accuracy and a high price, a determination of the position, orientation, etc., can be achieved with the same level of accuracy as with a three-dimensional measuring instrument, thus reducing the costs of implementing such a device.Furthermore, according to the present second example, a two-dimensional image and three-dimensional measurement data in the same area (capture area: a presence area of ​​a multitude of workpieces) are acquired, corrected, and calculated. For example, as in [PTL 1], listed in [CITATION LIST], in a capture method where the capture area of ​​a second image (a three-dimensional image containing three-dimensional point group data) is always smaller than the capture area (two-dimensional camera image) of a first image, three-dimensional measurement data for a workpiece in a portion of an area are not captured and therefore cannot be used to calculate a captured position of the workpiece in that area, and the positional accuracy decreases. This can be prevented by the present second example.

[0043] In this way, in the first and second examples of the position correction program (position correction device) according to the present embodiment, a two-dimensional image is captured with respect to a presence area of ​​a plurality of workpieces, and a three-dimensional measurement is also performed on the same area, and three-dimensional measurement data (for example, three-dimensional point group data) are acquired. By comparing the three-dimensional measurement data with the two-dimensional image, the detection accuracy of a position, orientation, outline, and the like of the workpiece can then be improved. It should be noted that, for example,With a low computational load, the first and second examples of the position correction program according to the embodiment described above can be executed by a computational processing unit in the robot controller 2 instead of a dedicated position correction device 3. In this case, the position correction device 3 is integrated into the robot controller 2. Conversely, if the computational effort for performing machine learning is high, the position correction device 3 can also be a host computer, a general-purpose computer, or the like, located at a dedicated workstation near the robot controller 2 or at a location isolated from the robot system 100. The position correction device 3 cannot perform the machine learning and can only process a learning result (e.g.,learned model data) is received, which was obtained by performing machine learning using another device and service, e.g., a cloud service where a GPU device can be used, and can perform the computational processing.

[0044] The position correction program according to the embodiment described above can be recorded and provided in a computer-readable, non-transferable recording medium and a non-volatile semiconductor memory, and it can be provided in a wired or wireless manner. For example, an optical disc such as a Compact Disc Read Only Memory (CD-ROM) or DVD-ROM, a hard drive, or similar device could serve as the computer-readable, non-volatile recording medium. Furthermore, a programmable read-only memory (PROM), flash memory, and the like could serve as the non-volatile semiconductor memory. Distribution from a server device could also be via a local area network (LAN) in a wired or wireless manner, or via a wide area network (WAN) such as the internet.

[0045] As described in detail above, the position correction device, the robot system and the position correction program according to the present embodiment can prevent the occurrence of misdetections and non-detection problems and improve the detection / detection accuracy of a workpiece (object).

[0046] Although the present disclosure has been described in detail above, it is not limited to the individual embodiments described above. Various types of additions, substitutions, modifications, partial deletions, and the like may be made to the embodiments without affecting the purpose of the present disclosure or the content described in the claims and the scope of the present disclosure resulting from their equivalents. Furthermore, the embodiments may also be carried out in combination. For example, a sequence of operations and a sequence of parts of the processing are given in the embodiments described above as an example, which is not limited thereto. The same applies if a numerical value or a numerical expression is used in the description of the embodiments described above.

[0047] With regard to the embodiments and variations described above, the following descriptions are further disclosed. [Annex 1]

[0048] Position correction device (3) which, based on a two-dimensional image (P1) in which a presence area of ​​a plurality of workpieces (D1 to D9, B0, D) is detected, detects at least one detected position of at least one of the workpieces (B0, D) and corrects (B1, B11 to B13) the detected position of at least one of the workpieces (B0) and outputs, based on three-dimensional measurement data (P2) in which the presence area of ​​the plurality of workpieces (D1 to D9, B0, D) is measured. [Annex 2]

[0049] Position correction device according to Annex 1, wherein the position correction device (3) calculates a shape area (A1) of the workpiece (B0, D) in the two-dimensional image (P1) and, based on the calculated shape area (A1) of the workpiece (B0, D), calculates the detected position of at least one of the workpieces (B0) (B1, B11 to B13). [Annex 3]

[0050] Position correction device according to Annex 2, wherein the position correction device (3) calculates the shape area (A1) of the workpiece (B0, D) in the two-dimensional image (P1) using a learning result of machine learning. [Annex 4]

[0051] Position correction device according to Annex 2 or 3, wherein the position correction device (3) calculates the shape area (A1) of the workpiece (B0, D) in the two-dimensional image (P1) by performing image processing on the two-dimensional image (P1). [Annex 5]

[0052] Position correction device according to one of Annexes 2 to 4, wherein the position correction device (3) compares the three-dimensional measurement data (P2) with the two-dimensional image (P1), corrects the shape area (A1) of the workpiece (B0, D) in the two-dimensional image (P1) using a difference in a three-dimensional position contained in the three-dimensional measurement data (P2) (B1, B11 to B13) and outputs the detected position of the workpiece (B0) based on the corrected shape area. [Annex 6]

[0053] Position correction device according to one of Annexes 2 to 5, wherein the position correction device (3) compares the three-dimensional measurement data (P2) with the two-dimensional image (P1), calculates a feature based on the three-dimensional measurement data (P2), corrects the shape area of ​​the workpiece (B0, D) in the two-dimensional image (P1) using the calculated feature, and corrects and outputs the detected position of the workpiece (B0, D) based on the corrected shape area. [Annex 7]

[0054] Position correction device according to Annex 6, wherein the position correction device (3) is defined as the feature of at least one groove, one gap, one step, one three-dimensional planar surface or one three-dimensional curved surface. [Annex 8]

[0055] Position correction device (3) comprising calculating, based on three-dimensional measurement data (P2) in which a presence area of ​​a plurality of workpieces (D1 to D9, C0, D) is measured, at least one detected position of at least one of the workpieces (C0, D), and correcting (C1, C11 to C13) and outputting the detected position of at least one of the workpieces (C0) based on a two-dimensional image (P1) in which the presence area of ​​the plurality of workpieces (D1 to D9, C0, D) is detected. [Annex 9]

[0056] Position correction device according to Annex 8, wherein the position correction device (3) calculates a shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2), and based on the calculated shape area (A1) of the workpiece (C0, D) calculates the detected position of at least one of the workpieces (C0, D) (C1, C11 to C13). [Annex 10]

[0057] Position correction device according to Annex 9, wherein the position correction device (3) calculates the shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) using a learning result of machine learning. [Annex 11]

[0058] Position correction device according to Annex 9 or 10, wherein The position correction device (3) calculates the shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) by performing a processing of the three-dimensional measurement data (P2). [Annex 12]

[0059] Position correction device according to one of Annexes 9 to 11, wherein the position correction device (3) compares the two-dimensional image (P1) with the three-dimensional measurement data (P2), corrects the shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) using a difference in a pixel value contained in the two-dimensional image (P1), and corrects and outputs the detected position of the workpiece (C0, D) based on the corrected shape area (A1). [Annex 13]

[0060] Position correction device according to one of Annexes 9 to 12, wherein the position correction device (3) compares the two-dimensional image (P1) with the three-dimensional measurement data (P2), calculates a feature based on the two-dimensional image (P1), corrects the shape area (A1) of the workpiece (C0, D) in the three-dimensional measurement data (P2) using the calculated feature, and corrects and outputs the detected position of the workpiece (C0, D) based on the corrected shape area (A1). [Annex 14]

[0061] The position correction device according to Annex 13, wherein the position correction device (3) is calculated as the feature of at least one of an edge, a groove, a slot, a step, a circle, a planar surface, a curved surface and a pattern of feature points. [Annex 15]

[0062] Position correction device according to one of Annexes 1 to 14, wherein the position correction device (3) calculates the detected position of at least one of the workpieces (B0, C0, D) and also at least one position, outline and size of at least one of the workpieces (B0, C0, D). [Annex 16]

[0063] Position correction device according to one of Annexes 1 to 15, wherein the position correction device (3) corrects and outputs the detected position of at least one of the workpieces (B0, C0, D) as well as at least one position, outline and size of at least one of the workpieces (B0, C0, D). [Annex 17]

[0064] Robot system (100) with: a robot (1) that performs a predetermined task on a workpiece (B0, C0, D); a detection unit (4) that captures a two-dimensional image (P1) in which a presence area of ​​a plurality of workpieces (D1 to D9, B0, C0, D) is captured, and also captures three-dimensional measurement data (P2) about the presence area of ​​the plurality of workpieces (D1 to D9, B0, C0, D); a position correction device (3) which, based on the two-dimensional image (P1) and the three-dimensional measurement data (P2) from the acquisition unit (4), calculates at least one detected position of at least one of the workpieces (B0, C0, D) and also corrects and outputs the detected position of at least one of the workpieces (B0, C0, D); and a robot controller (2) that receives an output from the position correction device (3), issues a control command to the robot (1) and controls the robot (1), wherein the position correction device (3) is the position correction device according to one of Annexes 1 to 16. [Annex 18]

[0065] Robot system according to Annex 17, wherein the robot (1) is controlled so that it removes the workpiece (B11 to B13, C11 to C13) in the position detected by the position correction device (3), based on the control command issued by the robot control (2). [Annex 19]

[0066] Position correction program that causes an invoice processing unit to perform processing in which, based on a two-dimensional image (P1) in which a presence area of ​​a plurality of workpieces (D1 to D9, B0, D) is captured, at least one captured position of at least one of the workpieces (B0, D) is calculated, and the captured position of at least one of the workpieces (B0) is corrected and output, based on three-dimensional measurement data (P2) in which the presence area of ​​the plurality of workpieces (D1 to D9, B0, D) is measured. [Annex 20]

[0067] Position correction program that causes an invoice processing unit to perform processing in which, based on three-dimensional measurement data (P2) in which a presence area of ​​a plurality of workpieces (D1 to D9, C0, D) is measured, at least one detected position of at least one of the workpieces (C0, D) is calculated and the detected position of at least one of the workpieces (C0) is corrected (C1, C11 to C13) and output, based on a two-dimensional image (P1) in which the presence area of ​​the plurality of workpieces (D1 to D9, C0, D) is detected. REFERENCE MARK LIST 1 robot 2 Robot control 3 Position correction device 4. Unit of measurement (measuring unit) 10 robot mechanism unit 1 arm 12 End effector (handpiece) 100 robot systems A1 Form area A2 area B0, B1, B11 to B13; C0, C11 to C13; D1 to D9 Object (workpiece) QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2016-192135 A

[0004] JP 2010-120141 A

[0004]

Claims

[1] Position correction device comprising calculating, based on a two-dimensional image in which a presence area of ​​a plurality of workpieces is captured, at least one captured position of at least one of the workpieces, and correcting and outputting the captured position of at least one of the workpieces based on three-dimensional measurement data in which the presence area of ​​the multiple workpieces is measured. [2] Position correction device according to claim 1, wherein the position correction device calculates a shape area of ​​the workpiece in the two-dimensional image and calculates the detected position of at least one of the workpieces based on the calculated shape area of ​​the workpiece. [3] Position correction device according to claim 2, wherein the position correction device calculates the shape area of ​​the workpiece in the two-dimensional image using a learning result of machine learning. [4] Position correction device according to claim 2 or 3, wherein the position correction device calculates the shape area of ​​the workpiece in the two-dimensional image by performing image processing on the two-dimensional image. [5] Position correction device according to one of claims 2 to 4, wherein the position correction device compares the three-dimensional measurement data with the two-dimensional image, corrects the shape area of ​​the workpiece in the two-dimensional image using a three-dimensional position difference contained in the three-dimensional measurement data, and corrects and outputs the detected position of the workpiece based on the corrected shape area. [6] Position correction device according to one of claims 2 to 5, wherein the position correction device compares the three-dimensional measurement data with the two-dimensional image, calculates a feature based on the three-dimensional measurement data, corrects the shape area of ​​the workpiece in the two-dimensional image using the calculated feature, and corrects and outputs the detected position of the workpiece based on the corrected shape area. [7] Position correction device according to claim 6, wherein the position correction device comprises as a feature at least one of a groove, a gap, a step, a three-dimensional planar surface and a three-dimensional curved surface. [8] Position correction device comprising calculating, based on three-dimensional measurement data in which a presence area of ​​a plurality of workpieces is measured, at least one detected position of at least one of the workpieces, and correcting and outputting the detected position of at least one of the workpieces based on a two-dimensional image in which the presence area of ​​the multiple workpieces is detected. [9] Position correction device according to claim 8, wherein the position correction device calculates a shape area of ​​the workpiece in the three-dimensional measurement data and calculates the detected position of at least one of the workpieces based on the calculated shape area of ​​the workpiece. [10] Position correction device according to claim 9, wherein the position correction device calculates the shape area of ​​the workpiece in the three-dimensional measurement data using a learning result of machine learning. [11] Position correction device according to claim 9 or 10, wherein the position correction device calculates the shape area of ​​the workpiece in the three-dimensional measurement data by performing a processing of the three-dimensional measurement data. [12] Position correction device according to one of claims 9 to 11, wherein the position correction device compares the two-dimensional image with the three-dimensional measurement data, corrects the shape area of ​​the workpiece in the three-dimensional measurement data using a difference of a pixel value contained in the two-dimensional image, and corrects and outputs the detected position of the workpiece based on the corrected shape area. [13] Position correction device according to one of claims 9 to 12, wherein the position correction device compares the two-dimensional image with the three-dimensional measurement data, calculates a feature based on the two-dimensional image, corrects the shape area of ​​the workpiece in the three-dimensional measurement data using the calculated feature, and corrects and outputs the detected position of the workpiece based on the corrected shape area. [14] Position correction device according to claim 13, wherein the position correction device calculates as a feature at least one of an edge, a groove, a gap, a step, a circle, a planar surface, a curved surface and a pattern of feature points. [15] Position correction device according to one of claims 1 to 14, wherein the position correction device calculates the detected position of at least one of the workpieces and also at least one position, outline and / or size of at least one of the workpieces. [16] Position correction device according to one of claims 1 to 15, wherein the position correction device corrects and outputs the detected position of at least one of the workpieces and also at least one of the position, outline and size of at least one of the workpieces. [17] Robot system comprising: a robot that performs predetermined tasks on a workpiece; a capture unit that captures a two-dimensional image in which a presence area of ​​a plurality of workpieces is captured, and also captures three-dimensional measurement data about the presence area of ​​the plurality of workpieces; a position correction device that calculates at least one detected position of at least one of the workpieces based on the two-dimensional image and the three-dimensional measurement data from the detection unit, and also corrects and outputs the detected position of at least one of the workpieces; and a robot controller that receives an output from the position correction device, issues a control command to the robot, and controls the robot, wherein the position correction device is the position correction device according to one of claims 1 to 16. [18] Robot system according to claim 17, wherein the robot is controlled in such a way that it removes the workpiece in the detected position output by the position correction device based on the control command issued by the robot control. [19] Position correction program which causes an invoice processing unit to perform processing in which, based on a two-dimensional image in which a presence area of ​​a plurality of workpieces is captured, at least one captured position of at least one of the workpieces is calculated, and the captured position of at least one of the workpieces is corrected and output based on three-dimensional measurement data in which the presence area of ​​the multiple workpieces is measured. [20] Position correction program which causes an invoice processing unit to perform processing in which, based on three-dimensional measurement data in which a presence area of ​​a plurality of workpieces is measured, at least one detected position of at least one of the workpieces is calculated, the detected position of at least one of the workpieces is corrected and output, based on a two-dimensional image in which the presence area of ​​the multiple workpieces is detected.

Images