A training data generation device, a training data generation method, and a program.

Abstract

This training data generating device generates training data relating to an object, and is characterized by comprising: a three-dimensional data acquiring unit (106) that acquires three-dimensional data obtained by measuring the object using a three-dimensional measuring unit; an image data acquiring unit (107) that acquires image data obtained by imaging the object using an imaging unit; an estimating unit (200; 300; 400) that estimates the position of an estimation target part of the object on the basis of the three-dimensional data; an image position estimating unit (208; 308; 408) that estimates the position of the estimation target part in the image data on the basis of the estimated position of the estimation target part; and a storage control unit (110) that associates the image data with the position of the estimation target part in the image data, and stores the same as training data in a storage unit.

Description

Learning data generation device, learning data generation method, program, and measurement system

[0001] The present disclosure relates to a position estimation technology, and more particularly to a training data generation device, a training data generation method, a program, and a measurement system.

[0002] Patent Document 1 discloses an edge position detection device that detects the position of at least one edge included in a pattern element group in an image showing one linear pattern element facing a first direction on an object, or a pattern element group that is a plurality of linear pattern elements facing the first direction and arranged in a second direction perpendicular to the first direction, the edge position detection device including a profile acquisition unit that acquires a luminance profile in an intersecting direction that is parallel to the second direction and intersects with the pattern element group in the image showing the pattern element group on the object, and m recesses (m- a calculation unit that fits a model function that is symmetrical in the cross direction, the model function being a combination of m bell-shaped functions corresponding to the m concave portions and (m-1) bell-shaped functions corresponding to the (m-1) convex portions, to the luminance profile having 1) convex portions, while satisfying constraints based on design data of the group of pattern elements, and determines a plurality of coefficients included in the m bell-shaped functions and the (m-1) bell-shaped functions of the model function; and an edge position acquisition unit that obtains the position of the at least one edge based on the model function.

[0003] JP 2016-50794 A

[0004] According to the technology disclosed in Patent Document 1, which detects edges by fitting a model function to the brightness profile of an image of an object, there is a problem in that the accuracy of edge detection varies depending on various conditions such as the position of the object, the lighting conditions when acquiring the image, and the texture of the object.

[0005] The present disclosure has been made to solve such problems, and aims to provide a position estimation technique that can better estimate the position of an estimation target portion of an object.

[0006] A learning data generation device according to an embodiment of the present disclosure is a learning data generation device that generates learning data related to an object, and is characterized by comprising: a three-dimensional data acquisition unit that acquires three-dimensional data measured by a three-dimensional measurement unit; an image data acquisition unit that acquires image data of the object captured by an imaging unit; an estimation unit that estimates the position of an estimation target portion of the object based on the three-dimensional data; an image position estimation unit that estimates the position of the estimation target portion on the image data based on the estimated position of the estimation target portion; and a memory control unit that associates the image data with the position of the estimation target portion on the image data and stores them in a memory unit as learning data.

[0007] According to the learning data generation device according to the embodiment of the present disclosure, the position of the estimation target portion of the object can be estimated more accurately.

[0008] 6A is a diagram showing the configuration of a training data generation device. FIG. 6B is a diagram showing an example of the configuration of a training data generation unit included in the training data generation device. FIG. 6C is a diagram for explaining the difficulty of edge estimation. FIG. 3A is an image of a workpiece having a machining edge. FIG. 3B is a graph showing pixel values ​​when the Y coordinate of the image in FIG. 3A is 200. FIG. 6B is a flowchart showing a training data generation method performed by the training data generation device. FIG. 6A is a flowchart showing an example of a model fitting method performed by the training data generation unit. FIG. 6B is an explanatory diagram showing the model fitting method performed by the training data generation unit and the projection of an estimation target portion. FIG. 6A is a diagram showing the distribution of a measured three-dimensional point cloud. FIG. 6B is a diagram showing the fitting of a function to a measured three-dimensional point cloud to estimate a straight line. FIG. 6C is a diagram showing an image onto which the estimated straight line is projected. FIG. 6B is a diagram showing an example of the configuration of a training data generation unit included in the training data generation device. FIG. 6A is a flowchart showing an example of the model fitting method performed by the training data generation unit. FIG. 6B is an explanatory diagram showing the model fitting method performed by the training data generation unit. Fig. 1 is a flowchart showing a method for calculating measurement positions performed by a measurement point setting unit. Fig. 2 is an explanatory diagram showing the state of the method for calculating measurement positions performed by a measurement point setting unit. Fig. 3 is a diagram showing the configuration of a measurement system. Fig. 4 is a diagram showing an example of the hardware configuration of a stage control unit. Fig. 5 is a diagram showing an example of the hardware configuration of a stage control unit.

[0009] Various embodiments of the present disclosure will be described in detail below with reference to the drawings. In the drawings, identical or similar parts are designated by identical or similar reference numerals, and redundant explanations of such parts will be omitted. In addition, in this disclosure, the term "or" is used to mean an inclusive logical OR unless otherwise specified.

[0010] Embodiment 1. <Configuration> A training data generation device 100 according to the present disclosure will be described with reference to FIG. 1 . As shown in FIG. 1 , the training data generation device 100 is connected to a 3D measurement unit 101 and an imaging unit 102 so as to be able to exchange data. In one aspect, the training data generation device 100 is a device for estimating a target portion, such as a machined edge, on an image of an object having a three-dimensional shape more reliably than conventional techniques. Examples of the object include a workpiece to be machined, but also include any object having a three-dimensional shape. To achieve this objective, the training data generation device 100 not only acquires image data from the imaging unit 102 as in conventional techniques, but also acquires three-dimensional data from the 3D measurement unit 101.

[0011] Here, problems with the conventional technology will be described with reference to FIGS. 3A and 3B. FIG. 3A is an image of a workpiece having a machining edge. In FIG. 3A, the thick line extending from the lower left to the upper right of the figure is the machining edge. FIG. 3B is a graph showing pixel values ​​at a Y coordinate of 200 in the image of FIG. 3A. Generally, when detecting an edge through image processing, a portion with a large change or gradient in pixel value is detected as an edge. Therefore, when the image contains multiple or numerous portions with large changes in pixel value, as in the image of FIG. 3A, a portion that is not a machining edge may be erroneously detected as an edge. The training data generation device 100 of the present disclosure aims to estimate the position of the machining edge while suppressing such erroneous detection.

[0012] 1 , the image capturing unit 102 is realized by a camera equipped with an image capturing element such as a CMOS sensor capable of capturing two-dimensional brightness information of an object. The image capturing unit 102 supplies image data of the captured object to the training data generation device 100.

[0013] The three-dimensional measurement unit 101 is a measurement device capable of measuring the three-dimensional position of an object. The three-dimensional measurement unit 101 may measure the three-dimensional position of an object by contacting the object or by not contacting the object. For example, the three-dimensional measurement unit 101 is realized by a three-dimensional measurement sensor capable of contactlessly measuring the three-dimensional position of an object to be sensed, such as a frequency-modulated continuous wave (FMCW) Lidar (Light Detection and Ranging) sensor. The three-dimensional measurement unit 101 supplies the measured three-dimensional data of the object to the training data generation device 100.

[0014] 1, the training data generation device 100 includes an input I / F 103-1, a three-dimensional data acquisition unit 106, a three-dimensional point cloud data generation unit 108, an input I / F 103-2, an image data acquisition unit 107, a training data generation unit 109, a storage control unit 110, and a storage unit 104. The three-dimensional data acquisition unit 106, the three-dimensional point cloud data generation unit 108, the image data acquisition unit 107, the training data generation unit 109, and the storage control unit 110 are realized by a processor 105 serving as a computer. The processor 105 realizes the functions of the three-dimensional data acquisition unit 106, the three-dimensional point cloud data generation unit 108, the image data acquisition unit 107, the training data generation unit 109, and the storage control unit 110 by reading and executing programs stored in the storage unit 104. The storage unit 104 is realized by a memory. Examples of memory include non-volatile or volatile semiconductor memory such as RAM (random access memory), ROM (read-only memory), flash memory, EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.

[0015] (Input I / F) The input I / F 103-1 is an interface device that receives three-dimensional data from the three-dimensional measurement unit 101. The input I / F 103-1 transmits the received three-dimensional data to the three-dimensional data acquisition unit 106. The input I / F 103-2 is an interface device that receives image data from the three-dimensional measurement unit 101. The input I / F 103-2 transmits the received image data to the image data acquisition unit 107.

[0016] (Three-Dimensional Data Acquisition Unit) The three-dimensional data acquisition unit 106 is a functional unit that acquires three-dimensional data from the input I / F 103-1 and supplies the acquired three-dimensional data to the three-dimensional point cloud data generation unit 108.

[0017] (Three-dimensional point cloud data generation unit) The three-dimensional point cloud data generation unit 108 is a functional unit that generates three-dimensional point cloud data from the three-dimensional data supplied from the three-dimensional data acquisition unit 106. The three-dimensional data of multiple positions obtained by a single measurement by the three-dimensional measurement unit 101 may be used as the three-dimensional point cloud data. The three-dimensional point cloud data generation unit 108 supplies the generated three-dimensional point cloud data to the training data generation unit 109. Hereinafter, the three-dimensional point cloud data supplied by the three-dimensional point cloud data generation unit 108 to the training data generation unit 109 will be referred to as three-dimensional point cloud data 201.

[0018] (Image Data Acquisition Unit) The image data acquisition unit 107 is a functional unit that acquires image data from the input I / F 103-2 and supplies the acquired image data to the learning data generation unit 109. Hereinafter, the image data that the image data acquisition unit 107 supplies to the learning data generation unit 109 will be referred to as image data 210.

[0019] (Learning Data Generation Unit) In one aspect, the learning data generation unit 109 is a functional unit that estimates the position of an estimation target portion of an object on image data 210 supplied from the image data acquisition unit 107, using the 3D point cloud data 201 generated by the 3D point cloud data generation unit 108. In another aspect, the learning data generation unit 109 is a functional unit that associates the estimated position of the estimation target portion with the image data 210, and generates, as learning data, the image data 210 associated with the position of the estimation target portion. To achieve this function, as shown in FIG. 2 , the learning data generation unit 109 includes an estimation unit 200, an image position estimation unit 208, and a calibration data storage unit 209.

[0020] (Estimation Unit) The estimation unit 200 is a functional unit that estimates the position of an estimation target portion of an object based on three-dimensional point cloud data. The estimation target portion of an object includes curved or straight portions of the object. The estimation target portion of an object includes portions where the object has been processed.

[0021] To achieve this function, as an example, the estimation unit 200 detects two adjacent measurement points in the three-dimensional point cloud data 201, at which the measurement distance along the z-axis direction changes along the x-axis direction by more than a predetermined threshold, and estimates the average value of the x-coordinates of the two detected measurement points as the x-coordinate of the estimation target portion of the object. The estimation unit 200 may estimate a curved portion or a straight portion of the estimation target portion by performing this operation while scanning in the y-axis direction.

[0022] As another example, the estimation unit 200 may classify each point included in the three-dimensional point cloud data into one of two clusters: a cluster having a first distance in the z-axis direction, and a cluster having a second distance in the z-axis direction that is different from the cluster having the first distance, and estimate the boundary between the clusters as a curved or straight line portion of the part to be estimated.

[0023] As another example, the estimation unit 200 may estimate a curved portion or a straight portion of the portion to be estimated by fitting a function to the three-dimensional point cloud data 201. When performing estimation by function fitting, the estimation unit 200 includes a model estimation unit 202 including a cost calculation unit 203 and a parameter update unit 204, and a position estimation unit 207, as shown in FIG.

[0024] (Model Estimation Unit) The model estimation unit 202 fits a prepared model function to the 3D point cloud data 201 to estimate geometric transformation parameters 205 and function parameters 206. When there are multiple model functions, the model function to be fitted may be selected by the user of the device or by the device itself. When selected by the device, a model function that reduces the cost calculated by the cost calculation unit 203 may be selected. The geometric transformation parameters 205 are parameters that represent geometric transformations such as affine transformations or homography transformations, and determine the position of the function to be used. The function parameters 206 are parameters that determine the shape of the function to be used.

[0025] The cost calculation unit 203 calculates the cost of the cost function from the 3D point cloud data 201 and the values ​​of the geometric transformation parameters 205 and function parameters 206. The parameter update unit 204 calculates parameters so as to reduce the cost calculated by the cost calculation unit 203, and updates the values ​​of the geometric transformation parameters 205 and function parameters 206.

[0026] (Position Estimation Unit) The position estimation unit 207 estimates the position of an estimation target portion of an object, such as a straight line on a three-dimensional curved surface, based on the values ​​of the geometric transformation parameters 205 and the function parameters 206 after being updated by the model estimation unit 202. The position estimation unit 207 supplies the estimated position of the estimation target portion to the image position estimation unit 208.

[0027] (Image Position Estimation Unit) The image position estimation unit 208 is a functional unit that estimates the position of the estimation target portion on the image data 210 based on the position of the estimation target portion supplied from the position estimation unit 207. Specifically, the image position estimation unit 208 estimates the position of the estimation target portion on the image data 210 by projecting the position of the estimation target portion supplied from the position estimation unit 207 onto the coordinate system of the image data 210 using calibration data on the positional relationship between the three-dimensional measurement unit 101 and the imaging unit 102, which is stored in the calibration data storage unit 209. The image position estimation unit 208 supplies information indicating the estimated position of the estimation target portion on the image data 210 to the memory control unit 110.

[0028] Here, the functions performed by the learning data generation unit 109 will be described with reference to FIG. 6. FIG. 6 is an explanatory diagram showing a model fitting method performed by the learning data generation unit 109 and the projection of the estimation target portion. As an example, a case is assumed in which a three-dimensional point cloud of a workpiece W having a three-dimensional shape with steps such as a machined edge is measured, and a sigmoid function having a planar spread is fitted to the measured three-dimensional point cloud. Note that instead of the sigmoid function, any function may be used in accordance with the shape of the workpiece.

[0029] 6A is a diagram showing the distribution of the measured three-dimensional point cloud. Fig. 6A shows that the three-dimensional measurement unit 101 performed three-dimensional measurement of the area indicated by the dashed line from above the workpiece W, and obtained a point cloud (an example of three-dimensional point cloud data 201) which is a collection of three-dimensional measurement points indicated by ●.

[0030] As shown in Figure 6B, the learning data generation unit 109 fits a function to this three-dimensional point cloud. The function to be fitted can be an appropriate function depending on the shape of the workpiece W. For a workpiece W with a three-dimensional shape having steps as shown in Figure 6A, a sigmoid function with a planar spread can be used. In this case, in addition to the parameters of the sigmoid function (scale, width a, bias), a total of six parameters are optimized, including the parameters for the translation t in the x direction, the rotation amount θ around the origin, and the skew in the x direction, taking into account the positional deviation or rotation of the measurement target.

[0031] First, a two-dimensional point cloud consisting of x and y is transformed by skew, rotation, and translation using equation (1). More specifically, the edge portions of the point cloud are aligned with the line at x=0 by skew, rotation, and translation.

[0032] Next, the value of the sigmoid function is calculated from the x value of the transformed h(x, y) using the scale, width, and bias according to equation (2). The calculated value of g(x, y) represents the z coordinate value.

[0033] Next, the cost calculation unit 203 calculates the cost or residual represented by the absolute value symbol on the right side of equation (3) for each measurement point, and the parameter update unit 204 estimates parameters that minimize the sum of the values ​​of the loss function defined by equation (3). That is, the value of g(x, y) calculated by equation (2) and the z-coordinate value z of the measured three-dimensional measurement point p are used to calculate the cost or residual. ｐ The parameters are estimated so that the sum of the differences (residuals) between p and p is minimized. The parameters are estimated by optimizing using a nonlinear least squares method such as the Levenberg-Marquardt method. In equation (3), p represents a measurement point, and P represents a point cloud that is a set of measurement points p.

[0034] The "estimated straight line" shown in FIG. 6B can be obtained as a straight line passing through the x and y coordinates of the portion where the value of the Z coordinate of the fitted function changes.

[0035] 6C is a diagram showing an image onto which the estimated straight line is projected. The projection is performed by converting the estimated straight line into image coordinates using the calibration data stored in the calibration data storage unit 209, and superimposing the converted straight line on the image captured by the imaging unit 102.

[0036] 1 , the storage control unit 110 associates the image data 210 with the position of the estimation target portion on the image data 210, and stores the associated data as learning data in the storage unit 104. The association may be performed by projecting a straight line onto the image data 210, or may be performed as a data set consisting of a set of the image data 210, parameters of the estimated straight line, and calibration data.

[0037] <Operation> Next, the operation of the training data generation device 100 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing a training data generation method performed by the training data generation device 100. Fig. 5 is a flowchart showing an example of a model fitting method performed by the training data generation unit 109.

[0038] (Step ST101) In step ST101, the three-dimensional data acquisition unit 106 acquires three-dimensional data obtained by measuring the object by the three-dimensional measurement unit 101, and the three-dimensional point cloud data generation unit 108 generates three-dimensional point cloud data from the three-dimensional data acquired by the three-dimensional data acquisition unit 106. Also in step ST101, the image data acquisition unit 107 acquires image data obtained by capturing an image of the object by the imaging unit 102.

[0039] (Step ST102) In step ST102, the estimation unit 200 estimates the position of the estimation target portion of the object based on the three-dimensional data. As a more specific example, the model estimation unit 202 fits a prepared model function to the three-dimensional point cloud data 201 to estimate function parameters 206 and geometric transformation parameters 205. Details of the processing in step ST102 will be described with reference to FIG. 4.

[0040] (Step ST102; Step ST201) In step ST201, model estimation unit 202 initializes geometric transformation parameters 205 and function parameters 206 with arbitrary values. Following the processing of step ST201, model estimation unit 202 repeats the optimization loop processing of steps ST202 to ST204 until the loop processing termination condition in step ST204 is satisfied. Note that, taking into consideration that the measurement points of 3D measurement unit 101 contain noise, 3D points that become outliers during the optimization loop may be removed.

[0041] (Step ST102; Step ST202) In step ST202, the cost calculation unit 203 calculates the cost of the cost function from the three-dimensional point group data 201, the geometric transformation parameters 205, and the function parameters 206.

[0042] (Step ST102; Step ST203) In step ST103, the parameter update unit 204 calculates parameters so as to reduce the cost calculated in step ST202, and updates the geometric transformation parameters 205 and the function parameters 206.

[0043] (Step ST102; Step ST204) In step ST204, the model estimation unit 202 determines whether to continue the parameter update loop depending on the number of calculations or the parameter update width. For example, the model estimation unit 202 determines to end the parameter update loop when the parameter update has been performed a predetermined number of times. Alternatively, the model estimation unit 202 determines to end the parameter update loop when the magnitude of the parameter value update is equal to or less than a predetermined value. Once the update is completed, the process proceeds to step ST103 in FIG. 4.

[0044] (Step ST103) In step ST103, the position estimation unit 207 estimates the position of the estimation target portion of the object based on the three-dimensional data. As a more specific example, the position estimation unit 207 estimates the position of the estimation target portion represented by a straight line or a curve on a three-dimensional curved surface based on the geometric transformation parameters 205 and function parameters 206 calculated in step ST102.

[0045] (Step ST103-2) In step ST103-2, based on the estimated position of the estimation target portion, the image position estimation unit 208 estimates the position of the estimation target portion on the image data 210. For example, based on calibration data of the three-dimensional measurement unit 101 and the imaging unit 102, the image position estimation unit 208 estimates the position of the estimation target portion on the image data 210 by projecting the position of the estimation target portion onto the coordinate system of the image data 210.

[0046] (Step ST104) In step ST104, the storage control unit 110 associates the image data 210 with the position of the estimation target portion on the image data 210, and stores the associated data in the storage unit 104 as learning data.

[0047] <Effects> According to the learning data generation device 100 of the first embodiment described above, the position of the part to be estimated, such as the processed edge of the object, is estimated using not only the image data 210 of the object captured by the imaging unit 102 but also the three-dimensional data of the object obtained by the three-dimensional measurement unit 101. Therefore, the position of the part to be estimated can be estimated better than in the conventional technology in which the position is estimated by image processing of the image data using only the image data.

[0048] Conventionally, when generating a learning model through machine learning using training data (also referred to as teacher data or correct labels) indicating the position of an estimation target portion such as a processing edge, it was necessary to prepare a huge amount of training data. Because high accuracy was required for the training data, experts created the training data by viewing images and performing annotation work. The training data generation device 100 of the present disclosure estimates the position of an estimation target portion such as an object's processing edge using not only the image data 210 of the object captured by the imaging unit 102 but also the 3D data of the object obtained by the 3D measurement unit 101, thereby enabling automatic generation of such training data. Therefore, the training data generation device 100 of the present disclosure eliminates the need for expert annotation work, which was previously required.

[0049] Embodiment 2. <Configuration> In the cost calculation in Embodiment 1, all 3D point clouds were treated equally in model fitting. When estimating an estimation target portion such as a processing edge, it is important to consider 3D points close to the estimation target portion such as a processing edge. By setting a high weight for these points, the accuracy of fitting to the 3D shape can be improved. Therefore, as Embodiment 2, a training data generation device 100 that can increase the weight for 3D points close to the estimation target portion will be described with reference to FIG. 1 and FIGS. 7 to 9. The overall configuration of the training data generation device 100 according to Embodiment 2 is similar to the overall configuration of the training data generation device 100 according to Embodiment 1 shown in FIG. 1, and therefore a redundant description will be omitted.

[0050] As shown in Figure 7, the learning data generation unit 109 provided in the learning data generation device 100 according to embodiment 2 includes an estimation unit 300, an image position estimation unit 308, and a calibration data storage unit 309, similar to the learning data generation unit 109 provided in the learning data generation device 100 according to embodiment 1.

[0051] The estimation unit 300 includes a model estimation unit 302 including a cost calculation unit 303 and a parameter update unit 304 , and a position estimation unit 307 , similar to the estimation unit 200 according to the first embodiment.

[0052] The estimation unit 300 according to the second embodiment determines whether a captured image represented by image data 310 or a plurality of image regions included in the captured image includes an estimation target portion such as a processed edge, and increases the cost weight for three-dimensional measurement points corresponding to the captured image or a plurality of image regions included in the captured image that are determined to include the captured image. By increasing the cost weight for such three-dimensional measurement points, the effect of optimization for such three-dimensional measurement points can be more strongly reflected, thereby improving the accuracy of function fitting to the estimation target portion.

[0053] To achieve such a function, estimation unit 300 differs from estimation unit 200 according to Embodiment 1 in that it further includes a weight calculation unit 311 and an image feature calculation unit 312. By including weight calculation unit 311 and image feature calculation unit 312, estimation unit 300 performs processing to change the weighting of the cost function based on the position of the estimation target portion estimated based on image data 310. Below, weight calculation unit 311 and image feature calculation unit 312 will be described in more detail.

[0054] (Image Feature Calculation Unit) The image feature calculation unit 312 is a functional unit that calculates an image feature amount to determine whether an area of ​​the captured image represented by the image data 310 includes a processed edge. The target for calculating the image feature amount may be the entire captured image represented by the image data 310, or an area smaller than the area of ​​the captured image represented by the image data 310.

[0055] To achieve such a function, as an example, the image feature calculation unit 312 cuts out a patch image of a size smaller than the size of the captured image from the captured image represented by the image data 310, and calculates image features for the patch image to determine whether the area contains a processed edge.

[0056] 9 , the image feature calculation unit 312 extracts a patch image PI1 or PI2 from the captured image represented by image data 310, and calculates the image feature amount of the patch image PI1 or PI2. The patch image is extracted by calculating image coordinates on the image corresponding to points in the 3D point cloud data 301 from the 3D point cloud data 301 and the calibration data stored in the calibration data storage unit 309, and extracting an area of ​​a predetermined size from the captured image centered on the calculated image coordinates. The size of the patch can be calculated depending on the resolution of the image and the density of the 3D point cloud data.

[0057] The image feature calculation unit 312 calculates, as the image feature, a statistical quantity such as the variance of pixel values ​​of a plurality of pixels included in the patch image. Note that the calculation of the image feature by the image feature calculation unit 312 is performed for all three-dimensional points included in the three-dimensional point cloud data 301.

[0058] As another example, a learning model that has learned the correlation between a patch image and the degree of proximity of the patch image to the processed edge may be stored in advance in a storage device (not shown), and the image feature calculation unit 312 may input the cut-out patch image to the learning model and obtain the degree of proximity of the cut-out patch image to the processed edge from the learning model as the image feature, thereby calculating the image feature. The learning model can be constructed using a deep neural network.

[0059] In this way, the image feature calculation unit 312 divides the image data 310 into multiple image regions based on the position of each three-dimensional point group contained in the three-dimensional data, and calculates the image feature amount for each of the multiple divided image regions.

[0060] (Weight Calculation Unit) The weight calculation unit 311 is a functional unit that calculates the weight of each point of the three-dimensional point cloud data 301 from the image feature amount calculated by the image feature calculation unit 312. For example, if the image feature amount is the variance of pixel values, the weight calculation unit 311 calculates the weight so that the larger the variance value, the larger the weight. Also, if the image feature amount is the degree of closeness to the processing edge, the weight calculation unit 311 calculates the weight so that the larger the value of closeness to the processing edge, the larger the weight. The weight calculation unit 311 normalizes the image feature amount of each three-dimensional measurement point to calculate the weight w of each point. ｐ Calculate.

[0061] In this way, the weight calculation unit 311 sets a larger weighting of the cost function for the first three-dimensional point, which is closer to the position of the estimation target part (processing edge) estimated based on the image data 310, than the second three-dimensional point, out of the first three-dimensional point and second three-dimensional point included in the three-dimensional data.

[0062] In addition, when the object for calculating the image feature is the entire captured image represented by the image data 310, the image feature calculation unit 312 may apply a Gaussian filter to the entire captured image to average and binarize the pixel values, calculate the position where the pixel value changes as the image feature, and the weight calculation unit 311 may weight the pixel value so that it is high at the position where the pixel value changes and decreases as it moves away from that position.

[0063] In the second embodiment, the cost calculation unit 303 calculates the cost using the following equation (4).

[0064] In the second embodiment, the cost defined by the formula (4) is optimized. The optimization has been explained in accordance with the first embodiment, so a duplicate explanation will be omitted.

[0065] <Operation> Next, the operation of the training data generation unit 109 included in the training data generation device 100 according to embodiment 2 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of a model fitting method performed by the training data generation unit 109. Note that the overall operation of the training data generation device 100 is the same as in Fig. 4, and therefore description thereof will be omitted.

[0066] (Step ST102; Step ST301) In step ST201, the model estimation unit 202 initializes the geometric transformation parameters 205 and the function parameters 206 to arbitrary values.

[0067] (Step ST102; Step ST302) In step ST302, image feature calculation unit 312 calculates corresponding image coordinates from 3D point cloud data 301 and the calibration data stored in calibration data storage unit 309, cuts out a patch image centered on the calculated image coordinates from the captured image represented by image data 310, and calculates image feature amounts for the patch image to determine whether the patch image is a region including a processed edge. This calculation is performed for all 3D points included in 3D point cloud data 301.

[0068] (Step ST102; Step ST303) In step ST303, the weight calculation section 311 normalizes the image feature amount of each three-dimensional point obtained in step ST302, and calculates the weight of each point.

[0069] (Step ST102; Step ST304) In step ST304, cost calculation section 303 calculates the cost of the cost function from 3D point cloud data 301, geometric transformation parameters 305, and function parameters 306. At this time, the weight calculated in step ST303 is used.

[0070] (Step ST102; Step ST305) In step ST305, the parameter update unit 304 calculates parameters so as to reduce the cost calculated in step ST304, and updates the geometric transformation parameters 305 and the function parameters 306.

[0071] (Step ST102; Step ST306) The model estimation unit 302 determines whether to continue the parameter update loop depending on the number of step calculations or the parameter update width. When the update is completed, the process proceeds to step ST103 in FIG.

[0072] <Effects> According to the training data generation device 100 according to the second embodiment described above, the image feature calculation unit 312 and the weight calculation unit 311 are provided, so it is possible to set large weights for 3D points that are close to the part to be estimated, such as a processed edge, etc. This makes it possible to improve the accuracy of fitting to the 3D shape.

[0073] As explained in accordance with the first embodiment, when estimating a target portion such as a processed edge by image processing, a portion that is not a processed edge may be erroneously detected as an edge. Therefore, when a patch image PI1 shown in Fig. 9 contains many portions where pixel values ​​change greatly, many edges will be erroneously detected when image processing is performed on the patch image PI1. By using a patch as in the second embodiment, an area with a certain extent can be evaluated, and therefore portions that appear to be processed edges can be identified globally.

[0074] 10 to 12 , a third embodiment will be described in which a measurement point setting unit 411 is added to the learning data generation unit 109 included in the learning data generation device 100 according to the first embodiment. Note that the measurement point setting unit 411 may be added to the configuration of the second embodiment. By providing the measurement point setting unit 411, it is possible to narrow the measurement interval near the estimation target portion, such as a straight line. Therefore, by performing additional measurements with narrow measurement intervals near the estimation target portion, it is possible to improve the accuracy of function fitting.

[0075] As shown in FIG. 10, the learning data generation unit 109 provided in the learning data generation device 100 according to embodiment 3 includes an estimation unit 400, an image position estimation unit 408, and a calibration data storage unit 409, similar to the learning data generation unit 109 provided in the learning data generation device 100 according to embodiment 1.

[0076] Furthermore, the learning data generation unit 109 included in the learning data generation device 100 according to the third embodiment includes a measurement point setting unit 411. The measurement point setting unit 411 is a functional unit that sets measurement points to be measured by the three-dimensional measurement unit 101 based on the acquired three-dimensional data.

[0077] <Operation> Next, the operation of the learning data generation device 100 will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a flowchart showing a method for calculating measurement positions performed by the measurement point setting unit 411, and Fig. 12 is an explanatory diagram showing the method for calculating measurement positions performed by the measurement point setting unit 411. Note that the flow shown in Fig. 11 is performed following step ST103 in Fig. 4.

[0078] (Step ST501) In step ST501, the measurement point setting unit 411 reads the measurement position coordinates of the three-dimensional point cloud data corresponding to the image v direction (see FIG. 12A), and converts the read measurement position coordinates into image coordinates using the calibration data stored in the calibration data storage unit 409.

[0079] (Step ST502) In step ST502, the measurement point setting unit 411 generates a sequence of points on the straight line calculated in step ST103 in the image width direction (image u direction), with the image coordinates corresponding to the coordinates in the v direction determined in step ST501 as the center. At this time, as shown in Fig. 12B, the measurement point setting unit 411 generates the sequence of points so that the spacing increases with increasing distance from the center. The measurement point setting unit 411 converts this sequence of points into the measurement coordinate system of the three-dimensional measurement unit 101 using the calibration data stored in the calibration data storage unit 409, and outputs the converted sequence of points to the measurement position database in the measurement position data storage unit 412.

[0080] <Effects> According to the third embodiment, it is possible to narrow the measurement intervals near the estimation target portion such as a straight line by providing the measurement point setting unit 411. Therefore, by performing additional measurements at narrower measurement intervals near the estimation target portion, it is possible to improve the accuracy of function fitting.

[0081] Embodiment 4. Next, a measurement system according to the present disclosure will be described as embodiment 4 with reference to Fig. 13 . Fig. 13 is a diagram showing the configuration of the measurement system according to embodiment 4. As shown in Fig. 13 , the measurement system includes a three-dimensional measuring device 501 that measures an object 506 and acquires three-dimensional data, an imaging unit 504 that images the object 506 and acquires image data, a stage 502 on which the object is placed and that moves the object 506 to change a measurement region in which the object 506 is measured by the three-dimensional measuring device 501, and a stage control unit 503 that sets measurement points to be measured by the three-dimensional measuring device 501 based on an estimated target portion of the object 506 estimated from the three-dimensional data, and controls the movement of the stage 502 based on the set measurement points.

[0082] The three-dimensional measuring device 501 is a device that corresponds to the three-dimensional measuring unit 101. The imaging unit 504 is a device that corresponds to the imaging unit .

[0083] The three-dimensional measuring device 501 and the imaging unit 504 are installed on an installation unit 505 so as to be able to measure or image an object 506 placed on a stage 502. The stage 502 is placed below the installation unit 505. As shown in Fig. 13 , the three-dimensional measuring device 501 or the imaging unit 504 measures or images the object 506 by moving the stage 502 on which the object 506 is placed.

[0084] The stage control unit 503 includes a setting unit 5031 that sets measurement points to be measured by the three-dimensional measuring device 501 based on an estimated target portion of the object 506 estimated from three-dimensional data measured by the three-dimensional measuring device 501. The estimated target portion may be a processed portion of the object 506. The stage control unit 503 controls the movement of the stage 502 based on the measurement points set by the setting unit 5031.

[0085] The setting unit 5031 may set the measurement points in the same manner as the measurement point setting unit 411 described in accordance with the third embodiment.

[0086] Next, an example of the hardware configuration of the stage control unit 503 will be described with reference to Figures 14A and 14B. The functions of the stage control unit 503 are realized by a processing circuitry. The processing circuitry may be a dedicated processing circuit 100a as shown in Figure 14A, or a processor 100b as a computer that executes a program stored in a memory 100c as shown in Figure 14B.

[0087] When the processing circuitry is a dedicated processing circuit 100a, the dedicated processing circuit 100a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

[0088] When the processing circuitry is the processor 100b, the functions of the stage control unit 503 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 100c. The processor 100b realizes the functions of the stage control unit 503 by reading and executing the programs stored in the memory 100c. Here, examples of the memory 100c include non-volatile or volatile semiconductor memory such as RAM (random access memory), ROM (read-only memory), flash memory, EPROM (erasable programmable read-only memory), and EEPROM (electrically erasable programmable read-only memory), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.

[0089] Instead of the processor 105 in the first to third embodiments, the functions performed by the processor 105 may be realized by the dedicated processing circuit 100a described with reference to FIG. 14A.

[0090] <Effects> The measurement system according to the fourth embodiment is provided with not only the imaging unit 504 that images the object 506 and acquires image data, but also the three-dimensional measuring device 501 that measures the object 506 and acquires three-dimensional data. Therefore, the position of the estimation target part, such as a processing edge, can be estimated better than in the prior art, which estimates the position of the estimation target part using only image data captured by the imaging device.

[0091] The measurement system according to the fourth embodiment also includes a stage control unit 503. The stage control unit 503 sets measurement points to be measured by the three-dimensional measurement device 501 based on the estimation target portion of the object 506 estimated using three-dimensional data measured by the three-dimensional measurement device 501, and controls the movement of the stage 502 based on the set measurement points. Therefore, a second measurement can be performed with a narrower measurement interval than in the first measurement for an area where an estimation target portion, such as an edge, is estimated to be located in the first measurement. By performing such a second measurement, detailed point cloud data of the estimation target portion can be obtained. By fitting a model to the data obtained in the second measurement, the fitting accuracy and the estimation accuracy of the estimation target portion are improved.

[0092] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate.

[0093] The learning data generation device of the present disclosure can be used as a device that generates learning data for performing machine learning.

[0094] 100 Learning data generation device, 100a Processing circuit, 100b Processor, 100c Memory, 101 3D measurement unit, 102 Imaging unit, 103 (103-1, 103-2) Input I / F, 104 Storage unit, 105 Processor, 106 3D data acquisition unit, 107 Image data acquisition unit, 108 3D point cloud data generation unit, 109 Learning data generation unit, 110 Storage control unit, 200 Estimation unit, 202 Model estimation unit, 203 Cost calculation unit, 204 Parameter update unit, 207 Position estimation unit, 208 Image position estimation unit, 209 Calibration data storage unit, 300 Estimation unit, 302 Model estimation unit, 303 Cost calculation unit, 304 Parameter update unit, 307 Position estimation unit, 308 Image position estimation unit, 309 Calibration data storage unit, 311 Weight calculation unit, 312 image feature calculation unit, 400 estimation unit, 408 image position estimation unit, 409 calibration data storage unit, 411 measurement point setting unit, 412 measurement position data storage unit, 501 three-dimensional measurement device, 502 stage, 503 stage control unit, 504 imaging unit, 505 installation unit, 5031 setting unit.

Claims

1. A learning data generation device that generates learning data related to an object, A 3D data acquisition unit acquires 3D data of the object measured by the 3D measurement unit, An image data acquisition unit acquires image data of the object captured by the imaging unit, An estimation unit that estimates the position of the estimated target portion of the object based on the three-dimensional data, An image position estimation unit that estimates the position of the estimated target portion on the image data based on the estimated position of the estimated target portion, The system includes a storage control unit that associates the image data with the position of the estimated portion on the image data and stores it in the storage unit as learning data. The estimation unit, Based on the three-dimensional data, a function to fit the target portion to be estimated is determined, and the target portion to be estimated is estimated based on the function. A cost function is calculated using the three-dimensional data, the geometric transformation parameters of the function, and the function parameters of the function. Based on the cost function, the geometric transformation parameters and the function parameters are updated to determine the function. Based on the position of the estimated portion estimated from the image data, the weighting of the cost function is modified. A learning data generation device characterized by the following features.

2. The estimated portion includes the straight portion of the object. The learning data generation device according to feature 1.

3. The estimated portion includes the portion of the object that has been processed. The learning data generation device according to feature 1.

4. The estimation unit sets a larger weight in the cost function for the first 3D point, which is closer to the position of the estimated portion estimated based on the image data, than the second 3D point, among the first 3D point and the second 3D point included in the 3D data. The learning data generation device according to feature 1.

5. The estimation unit divides the image data into multiple image regions based on the positions of each three-dimensional point cloud contained in the three-dimensional data, and estimates the position of the target portion based on the image features of each of the divided multiple image regions. The learning data generation device according to feature 1.

6. A learning data generation device that generates learning data related to an object, A 3D data acquisition unit acquires 3D data of the object measured by the 3D measurement unit, An image data acquisition unit acquires image data of the object captured by the imaging unit, An estimation unit that estimates the position of the estimated target portion of the object based on the three-dimensional data, An image position estimation unit that estimates the position of the estimated target portion on the image data based on the estimated position of the estimated target portion, A memory control unit that associates the image data with the position of the portion to be estimated on the image data and stores it in the memory unit as learning data, The system includes a measurement point setting unit that sets measurement points to be measured by the 3D measurement unit based on the acquired 3D data. A learning data generation device characterized by the following features.

7. The image position estimation unit estimates the position of the target portion on the image data using calibration data based on the positional relationship between the three-dimensional measurement unit and the imaging unit. A learning data generation device according to any one of claims 1 to 6.

8. A learning data generation method for generating learning data relating to an object, performed by a learning data generation device comprising a 3D data acquisition unit, an image data acquisition unit, an estimation unit, an image position estimation unit, and a memory control unit, wherein the learning data generation device generates learning data relating to an object, The three-dimensional data acquisition unit acquires three-dimensional data of the object measured by the three-dimensional measurement unit, The image data acquisition unit acquires image data of the object captured by the imaging unit, The estimation unit performs the steps of estimating the position of the target portion of the object based on the three-dimensional data, The image position estimation unit performs the step of estimating the position of the target portion on the image data based on the estimated position of the target portion, The memory control unit includes the step of associating the image data with the position of the portion to be estimated on the image data and storing it in the memory unit as learning data. The estimation unit, Based on the three-dimensional data, a function to fit the target portion to be estimated is determined, and the target portion to be estimated is estimated based on the function. A cost function is calculated using the three-dimensional data, the geometric transformation parameters of the function, and the function parameters of the function. Based on the cost function, the geometric transformation parameters and the function parameters are updated to determine the function. Based on the position of the estimated portion estimated from the image data, the weighting of the cost function is modified. A method for generating training data characterized by the following features.

9. A learning data generation method for generating learning data relating to an object, performed by a learning data generation device comprising a 3D data acquisition unit, an image data acquisition unit, an estimation unit, an image position estimation unit, a memory control unit, and a measurement point setting unit, wherein the learning data generation device generates learning data relating to an object, The three-dimensional data acquisition unit acquires three-dimensional data of the object measured by the three-dimensional measurement unit, The image data acquisition unit acquires image data of the object captured by the imaging unit, The estimation unit performs the steps of estimating the position of the target portion of the object based on the three-dimensional data, The image position estimation unit performs the step of estimating the position of the target portion on the image data based on the estimated position of the target portion, The memory control unit associates the image data with the location of the portion to be estimated on the image data and stores it in the memory unit as learning data. The measurement point setting unit includes the step of setting measurement points to be measured by the three-dimensional measurement unit based on the acquired three-dimensional data. A method for generating training data characterized by the following features.

10. A program to be executed by a computer in a learning data generation device that generates learning data related to an object, To the aforementioned computer, A process to acquire 3D data of the object measured by the 3D measurement unit, The process of acquiring image data of the object captured by the imaging unit, A process for estimating the position of the target portion of the object based on the aforementioned three-dimensional data, A process for estimating the position of the estimated target portion on the image data based on the estimated position of the target portion, A program for performing and executing a process of associating the image data with the position of the portion to be estimated on the image data and storing it in a memory unit as training data, The aforementioned estimation process is, Based on the three-dimensional data, a function to fit the target portion to be estimated is determined, and the target portion to be estimated is estimated based on the function. A cost function is calculated using the three-dimensional data, the geometric transformation parameters of the function, and the function parameters of the function. Based on the cost function, the geometric transformation parameters and the function parameters are updated to determine the function. This process includes modifying the weighting of the cost function based on the position of the estimated portion estimated based on the image data. program.

11. A program to be executed by a computer in a learning data generation device that generates learning data related to an object, To the aforementioned computer, A process to acquire 3D data of the object measured by the 3D measurement unit, The process of acquiring image data of the object captured by the imaging unit, A process for estimating the position of the target portion of the object based on the aforementioned three-dimensional data, A process for estimating the position of the estimated target portion on the image data based on the estimated position of the target portion, A process of associating the image data with the position of the portion to be estimated on the image data and storing it in the storage unit as training data, A program for performing the process of setting measurement points to be measured by the three-dimensional measurement unit based on the acquired three-dimensional data.