Three-dimensional shape measuring device and three-dimensional shape measuring method

Abstract

To correct the influences of distortion of an optical element with high accuracy.SOLUTION: Provided is a three-dimensional shape measuring device comprising: a virtual coordinate specification unit 402 for projecting feature points on a reference tool to an image plane of the imaging device and specifying the coordinates of virtual imaging pixels corresponding to the feature points on the image plane, using geometric characteristic information that represents the geometric characteristic of an imaging device and the relative position attitude information of the imaging device and the reference tool; a correction unit 403 for creating correction information for correcting the coordinates of measurement imaging pixels included in a measurement imaged image; and a shape specification unit 405 for correcting, on the basis of the geometric characteristic information and correction information, the coordinates of a plurality of measurement imaging pixels included in the measurement imaged image generated by the imaging device by imaging a measurement object and specifying the shape of the measurement object on the basis of measurement three-dimensional coordinates calculated based on the pixel coordinates.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a three-dimensional shape measuring apparatus and a three-dimensional shape measuring method for measuring the three-dimensional shape of a measurement object. [Background technology]

[0002] The light pattern projection method utilizes the principle of triangulation, in which a striped pattern is projected from a projector onto the object to be measured, and the pattern that deforms according to the shape of the object to be measured is photographed by a camera to perform three-dimensional shape measurement. In three-dimensional shape measurement, the measurement accuracy may be reduced due to the influence of image distortion caused by optical imperfections in the imaging unit or projection unit. For this reason, it is important to obtain image distortion by photographing a reference instrument for distortion correction, and to correct the image distortion. Patent Document 1 describes that image distortion is corrected by photographing a calibration pattern on a reference instrument. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-207346 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the method described in Patent Document 1, calibration is performed by irradiating light of a specific wavelength in order to ensure accuracy. However, since image distortion is complex, there is a problem that it is difficult to ensure sufficient accuracy with this method alone.

[0005] Therefore, the present invention has been made in consideration of these points, and has an object to provide a three-dimensional shape measuring device and a three-dimensional shape measuring method that can deal with complex image distortions and perform measurements with higher accuracy. [Means for solving the problem]

[0006] A three-dimensional shape measuring apparatus according to a first aspect of the present invention includes an imaging device, and measures the three-dimensional shape of a measurement object based on a measurement captured image of the measurement object captured by the imaging device. The three-dimensional shape measuring apparatus includes a measurement coordinate specifying unit that specifies coordinates of measurement captured pixels corresponding to feature points in a measurement captured image generated by capturing a reference tool having a plurality of feature points with the imaging device, and a measurement coordinate specifying unit that virtually projects the feature points on the reference tool onto an image plane of the imaging device using geometric characteristic information that indicates the geometric characteristics of the imaging device and relative position and orientation information between the imaging device and the reference tool. a correction unit that creates correction information for correcting the coordinates of the measurement imaging pixels included in the measurement imaging image based on a comparison between the coordinates of the actual imaging pixels and the coordinates of the virtual imaging pixels; and a shape specifying unit that corrects the coordinates of the measurement imaging pixels included in the measurement imaging image generated by the imaging device by imaging the measurement object based on the geometric characteristic information and the correction information, and specifies the shape of the measurement object based on measurement three-dimensional coordinates calculated based on the pixel coordinates.

[0007] The geometric characteristic information of the imaging device may include at least one of a function using a coefficient related to a radial distortion of an optical element and a function using a coefficient related to a circumferential distortion of the optical element. The correction unit may identify an error occurring between a coordinate of the virtual imaging pixel and a coordinate of the measured imaging pixel, the correction unit may create an error image in which the identified error is associated with a coordinate of a pixel corresponding to the error, and the shape identification unit may correct the coordinate of the measurement imaging pixel based on the error associated with the coordinate of a pixel corresponding to the measurement imaging pixel in the error image, and calculate the measurement three-dimensional coordinate based on the corrected coordinate.

[0008] The correction unit may estimate an error of a pixel to which an error is not associated based on a plurality of errors of a plurality of the measured captured pixels associated with coordinates different from the coordinates of a pixel to which the error is not associated, and generate the error image by associating the error with the coordinates of the pixel corresponding to the error.

[0009] The correction unit may further include a synthesis unit that creates a plurality of error images corresponding to a plurality of measured captured images generated by capturing images of the reference instrument installed in a plurality of positions and postures using the imaging device, and generates one or more averaged error images by averaging an error associated with the measurement captured pixel in one error image and an error associated with the same measurement captured pixel in another error image.

[0010] The correction unit may create the error images corresponding to the plurality of actual measurement images obtained by gradually rotating the reference instrument around an axis in the depth direction. The shape specification unit may obtain approximate values ​​of measurement three-dimensional coordinates corresponding to the measurement imaging pixels included in the measurement imaging images, and calculate the measurement three-dimensional coordinates based on errors specified using the plurality of error images, the depth direction coordinates of the positions of the reference instrument corresponding to the error images, and the depth direction coordinates of the approximate values. The three-dimensional shape measuring device may further include a projection device that projects a projection image, and the imaging device may generate the measurement imaging image by imaging the measurement object onto which the projection image is projected.

[0011] A three-dimensional shape measuring method according to a second aspect of the present invention is a three-dimensional shape measuring method for measuring the three-dimensional shape of a measurement object based on a measurement image obtained by capturing an image of the measurement object by an imaging device, and includes the steps of: identifying coordinates of actual measurement imaging pixels corresponding to feature points in a measurement image generated by capturing an image of a reference instrument having a plurality of feature points using the imaging device; virtually projecting feature points on the reference instrument onto an image plane of the imaging device using geometric characteristic information representing the geometric characteristics of the imaging device and relative position and orientation information between the imaging device and the reference instrument, and identifying coordinates of virtual imaging pixels corresponding to the feature points on the image plane; creating correction information for correcting coordinates of measurement imaging pixels included in the measurement imaging image based on a comparison between the coordinates of the actual measurement imaging pixels and the coordinates of the virtual imaging pixels; and correcting the coordinates of a plurality of measurement imaging pixels included in the measurement imaging image generated by the imaging device by capturing an image of the measurement object based on the geometric characteristic information and the correction information, and identifying the shape of the measurement object based on measurement three-dimensional coordinates calculated based on the pixel coordinates. Effect of the Invention

[0012] According to the present invention, it is possible to achieve an effect of enabling more accurate measurement in response to complex image distortion. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram for explaining an overview of a three-dimensional shape measuring apparatus according to an embodiment; [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a three-dimensional shape measuring apparatus. [Diagram 3] FIG. 13 is a diagram showing an example of feature points of a reference instrument. [Figure 4] FIG. 13 is a diagram showing the measurement of a reference instrument. [Diagram 5] 11 is a diagram for explaining a method for identifying the coordinates of a virtual imaging pixel. FIG. [Figure 6] 11A and 11B are diagrams illustrating an example of an error between an actually measured imaging pixel and a virtual imaging pixel. [Figure 7]11A to 11C are diagrams illustrating examples of types of projection images projected by a shape specifying unit. [Figure 8] 1 shows an example of a grayscale stripe pattern having a sinusoidal luminance distribution. [Figure 9] An example of a Gray code corresponding to a binary stripe pattern is shown below. [Figure 10] FIG. 13 is a diagram illustrating the relationship between absolute projected coordinates and relative projected coordinates. [Figure 11] 13 is a diagram showing a correspondence relationship between a first measurement imaging pixel and a second measurement imaging pixel. FIG. [Figure 12] 4 is a flowchart showing a processing procedure for generating an error image by the three-dimensional shape measuring device. [Figure 13] 4 is a flowchart showing a processing procedure for identifying the shape of a measurement object by the three-dimensional shape measuring device. [Figure 14] 13 is a diagram showing the relationship between the depth coordinate and the horizontal component of the error in the coordinate of a specific measured captured pixel. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Overview of the three-dimensional shape measuring device 100] 1(a) to 1(c) are diagrams for explaining an overview of a three-dimensional shape measuring apparatus 100 according to this embodiment. FIG. 1(a) shows the configuration of the three-dimensional shape measuring apparatus 100. The three-dimensional shape measuring apparatus 100 measures the three-dimensional shape of a measurement object based on a measurement captured image obtained by capturing an image of the measurement object by one or more imaging devices. The three-dimensional shape measuring apparatus 100 includes a first imaging device 1, a second imaging device 2, and a projection device 3 as optical devices. The three-dimensional shape measuring apparatus 100 includes a control unit 4 that controls various operations of these optical devices.

[0015] The projection device 3 is a projection device having a light source such as a light emitting diode or a laser. The projection device 3 projects a projection image including a pattern for identifying the projection coordinates onto the measurement surface of the measurement object. The projection coordinates indicate the positions of the projection pixels constituting the projection image projected by the projection device 3. The projection coordinates may be one-dimensional coordinates indicating either the vertical or horizontal position of the projection image, or may be two-dimensional coordinates indicating both the vertical and horizontal positions of the projection image. The pattern is, for example, a stripe pattern. In addition, the number of projection devices 3 is not limited to one, and the three-dimensional shape measuring device 100 may be equipped with any number of projection devices.

[0016] The first imaging device 1 has an optical element 11 and an imaging element 12. The optical element 11 is, for example, a lens. The optical element 11 may include a plurality of lenses and mirrors. When the projection device 3 projects a projection image onto the measurement object, the first imaging device 1 captures the projection image projected onto the measurement object to generate a first captured image. The first imaging device 1 is disposed such that its optical axis forms a predetermined angle with the optical axis of the projection device 3.

[0017] The second imaging device 2 has an optical element 21 and an imaging element 22. The optical element 21 is, for example, a lens. The optical element 21 may include a plurality of lenses and mirrors. When the projection device 3 projects a projection image onto the measurement object, the second imaging device 2 captures the projection image projected onto the measurement object to generate a second captured image. The second imaging device 2 is disposed so that its optical axis forms a predetermined angle with the optical axis of the projection device 3. The optical axis of the second imaging device 2 may be in the same plane as the optical axis of the first imaging device 1 and the optical axis of the projection device 3, but is not limited to this. The control unit 4 can be realized by, for example, a computer. The control unit 4 measures the three-dimensional shape of the measurement object based on the plurality of captured images generated by the first imaging device 1 and the second imaging device 2. In addition, the number of imaging devices is not limited to two, and the three-dimensional shape measuring device 100 may be provided with any number of imaging devices.

[0018] 1(b) and (c) show examples of measurement capture images generated by the first image capture device 1 in a state where the projection device 3 projects a projection image onto the measurement object. As shown in FIG. 1(b) and (c), the projection device 3 projects a projection image including a binary stripe pattern onto the measurement object. The binary stripe pattern is composed of a light projection area where light is projected and a non-projection area where light is not projected. FIG. 1(b) shows a measurement capture image generated by the first image capture device 1 and the second image capture device 2 when the projection device 3 projects a projection image including a binary stripe pattern onto a measurement surface that is smooth. The white area indicates the light projection area, and the black area indicates the non-projection area. When the measurement surface is smooth, the binary stripe pattern of the measurement capture image generated by the first image capture device 1 and the second image capture device 2 roughly matches the shape of the binary stripe pattern of the projection image.

[0019] FIG. 1(c) shows a measurement capture image generated by the first imaging device 1 or the second imaging device 2 when the projection device 3 projects a binary stripe pattern onto a measurement surface having a convex portion. In the measurement capture image of FIG. 1(c), a part of the image of the binary stripe pattern is in a deformed state. In the measurement capture image, the image of the binary stripe pattern is deformed by an amount according to the height of the convex portion. Thus, the three-dimensional shape measuring device 100 can measure the shape of the measurement object by identifying the height of each position of the convex portion based on the amount of deformation of the image of the binary stripe pattern in the measurement capture image.

[0020] The three-dimensional shape measuring device 100 stores in a storage unit parameters such as focal length and image center that represent the geometric characteristics of the first image capturing device 1 and the second image capturing device 2, geometric characteristic information that represents a function of lens distortion, and information on the relative position and orientation of each image capturing device and a reference instrument, and specifies the shape of the measurement object while correcting the distortion of the captured image by referring to this information. The three-dimensional shape measuring device 100 deals with errors in the measurement of three-dimensional coordinates that are difficult to deal with only by the geometric characteristic information that represents the geometric characteristics, by capturing an image of a feature point of the reference instrument whose two-dimensional or three-dimensional coordinates are known in advance. The three-dimensional shape measuring device 100 measures the three-dimensional coordinates of the measurement point of the measurement object by referring to the identified errors, and therefore can correct the influence of image distortion with higher accuracy.

[0021] [Configuration of 3D shape measuring device] 2 is a diagram showing the configuration of three-dimensional shape measuring apparatus 100. Three-dimensional shape measuring apparatus 100 includes a storage unit 5 in addition to first imaging device 1, second imaging device 2, projection device 3, and control unit 4 shown in FIG.

[0022] The storage unit 5 includes storage media such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 5 stores a program executed by the control unit 4. The control unit 4 is, for example, a CPU (Central Processing Unit). The control unit 4 executes the programs stored in the storage unit 5 to function as an actual coordinate specifying unit 401, a virtual coordinate specifying unit 402, a correction unit 403, a synthesis unit 404, and a shape specifying unit 405.

[0023] [Measurement of reference equipment] The actual measurement coordinate identifying unit 401 generates an actual measurement image by capturing an image of a reference instrument having a plurality of feature points using the first imaging device 1 or the second imaging device 2. The actual measurement coordinate identifying unit 401 identifies the coordinates of pixels (hereinafter also referred to as actual measurement pixels) corresponding to the feature points of the reference instrument in the generated actual measurement image.

[0024] 3(a) and 3(b) are diagrams showing examples of feature points of a reference instrument. Both of FIG. 3(a) and FIG. 3(b) show the reference instrument viewed from above. FIG. 3(a) shows a reference instrument in which a plurality of black circles are aligned vertically and horizontally. FIG. 3(b) shows a reference instrument with a checkered pattern. When the reference instrument shown in FIG. 3(a) is used, the actual coordinate determination unit 401 determines the center of the black circle as a feature point and determines the coordinates of the measured captured pixel corresponding to this feature point. When the reference instrument shown in FIG. 3(b) is used, the actual coordinate determination unit 401 determines each vertex of the white and black squares constituting the checkered pattern as a feature point and determines the coordinates of the measured captured pixel corresponding to these determined feature points.

[0025] The positions of the feature points of the reference instrument are assumed to have been measured in advance by another measuring instrument (not shown). For example, relative coordinate information indicating the relative coordinates of the other feature points, with one of the feature points included in the reference instrument set as the origin, is stored in the storage unit 5. Information indicating the coordinates of the multiple feature points included in the reference instrument may also be stored in the storage unit 5.

[0026] FIG. 4 is a diagram showing how the reference instrument 51 is measured. In the example of FIG. 4, the reference instrument 51 is a flat plate-shaped member, but is not limited thereto, and an arbitrary shape can be used as the reference instrument. The actual coordinate identification unit 401 captures an image of the reference instrument 51 one or more times. The reference instrument 51 is sequentially placed at a plurality of positions having different coordinates in the depth direction as indicated by the arrow in FIG. 4 during the measurement. The reference instrument 51 may also be placed in a plurality of different postures.

[0027] Furthermore, a plurality of positions at which the reference instrument 51 is installed and the orientation of the reference instrument 51 when installed at each position may be determined in advance. In this case, the positions and orientations may be approximate. When capturing an image of the reference instrument 51 and acquiring geometric characteristic information indicating the geometric characteristics of the imaging device, the actual coordinate determination unit 401 also acquires position and orientation information indicating the relative position and orientation between the imaging device and the reference instrument 51. The actual coordinate determination unit 401 can also measure the three-dimensional coordinates of each feature point on the reference instrument 51 by triangulation using a combination of imaging devices or a combination of an imaging device and a projection device, and acquire position and orientation information indicating the position and orientation of the reference instrument 51.

[0028] The actual coordinate specifying unit 401 generates a plurality of actual captured images by capturing an image of the reference instrument 51 when the reference instrument 51 is installed at each position and posture. The actual coordinate specifying unit 401 specifies the coordinates of the actual captured pixels of the feature points of the reference instrument 51 at each position, etc., based on the plurality of actual captured images.

[0029] Position and orientation information corresponding to the position and orientation of the reference instrument 51 is g (=1, . . . , G). Feature point information corresponding to the feature points of the reference instrument 51 is h (=1, . . . , H). If the position of a measured captured pixel on a captured image is expressed in homogeneous coordinates, the measured coordinate specification unit 401 determines the homogeneous coordinates of the measured captured pixel corresponding to the feature point h of the reference instrument 51. TIFF0007649145000001.tif57 is identified by the following formula (1).

number

[0030] [Identifying the coordinates of the virtual captured image] The virtual coordinate identifying unit 402 reads out, from the storage unit 5, geometric characteristic information representing the geometric characteristics of the first imaging device 1 or the second imaging device 2, and position and orientation information representing the relative position and orientation of the reference instrument 51 as viewed from the first imaging device 1 or the second imaging device 2. Using parameters and functions included in the read out geometric characteristic information and the position and orientation of the reference instrument 51 included in the position and orientation information, the virtual coordinate identifying unit 402 identifies the coordinates of an imaging pixel (hereinafter also referred to as a virtual imaging pixel) corresponding to a feature point on an image plane of the first imaging device 1 or the second imaging device 2, assuming that the feature point on the reference instrument 51 is virtually projected onto the image plane of the first imaging device 1 or the second imaging device 2.

[0031] FIG. 5 is a diagram for explaining a method for identifying the coordinates of a virtual imaging pixel. FIG. 5 shows a camera coordinate system with the optical center C of the first imaging device 1 as the origin, and a world coordinate system with the feature point O on the reference instrument 51 as the origin. The X-axis, Y-axis, and Z-axis of the camera coordinate system, and the Xw-axis, Yw-axis, and Zw-axis of the world coordinate system are shown. The Xw-axis and Yw-axis are coordinate axes indicating a plane on which a plurality of feature points are arranged on the reference instrument 51. The Zw-axis is a coordinate axis perpendicular to the plane on which a plurality of feature points are arranged, and corresponds to the depth direction shown in FIG. 4 in the example of this embodiment. f in FIG. 5 indicates the focal length of the optical element 11. Note that, following the convention, the image plane is shown arranged on the side where the imaging object exists.

[0032] The feature point M of the reference fixture 51 is expressed as M = [X W Y W Z W ] T ...(2). The virtual image pixel m obtained by projecting the feature point M onto the image plane of the first image capture device 1 is expressed as m = [uv] T ...(3) which includes the horizontal component u and the vertical component v of the image plane.

[0033] When the feature point M and the virtual captured pixel m are expressed in homogeneous coordinates, they are expressed by the following equations (4) and (5), respectively.

number

number

number

number

[0034] Among the components of matrix A shown in equation (8), u0 represents the horizontal component at the center of the captured image. v0 represents the vertical component at the center of the captured image. α represents the focal length f and the number of pixels per unit length in the horizontal direction k. u Using and, α = fk u ...(9) β is the focal length f and the number of pixels per unit length in the vertical direction k v Using and, β = fk v ...(10) In addition, although the equation (8) omits the shear coefficient of the image, it may be included.

[0035] Normalizing both sides of equation (7) gives the following equation (11).

number

[0036] When equations (5), (11), and (12) are rewritten taking into account the image distortion in the first imaging device 1, the following equations (13) to (15) are obtained.

number

number

[0037] In formula (16), r n is r n 2 =x n 2 +y n 2 ...(17) In addition, k i (i=1,2) is the coefficient related to the radial distortion of the optical element. p i (i=1,2) is the coefficient related to the circumferential distortion. x d The homogeneous coordinates of a given function f a Using this, it is expressed by the following equation (18).

number

[0038] Similarly, equation (12) can be expressed as b This is transformed into the following equation (19):

number

[0039] The virtual coordinate specification unit 402 determines the rotation matrix R g and translation vector T g Using the above, the homogeneous coordinates of the virtual captured pixel m corresponding to the feature point M having the position and orientation information g are specified using the following equation (20).

number

[0040] The virtual coordinate specification unit 402 specifies the homogeneous coordinates of the actual image capture pixels. The variables A, k, p, and R in equation (20) are set so that the error between TIFF0007649145000018.tif57 and the coordinates of the virtual captured pixel m is minimized. g , T g The obtained variables are stored in the memory unit 5.

[0041] [Identifying the error] The correction unit 403 compares the coordinates of the actual captured pixel identified by the actual coordinate identification unit 401 with the coordinates of the virtual captured pixel calculated using equation (20). The correction unit 403 creates correction information for correcting the coordinates of the captured pixel based on the comparison result.

[0042] FIG. 6 is a diagram showing an example of an error between a measured captured pixel and a virtual captured pixel. In FIG. 6, i indicates the horizontal coordinate in the measured captured image, and j indicates the vertical coordinate in the measured captured image. Measured captured pixels are indicated by circles, and virtual captured pixels are indicated by triangles. After determining each variable in equation (20), the correction unit 403 specifies an error occurring between the coordinates of the virtual captured pixel and the coordinates of the measured captured pixel. The error e between the measured captured pixel and the virtual captured pixel is g,h The coordinates of the measured image pixels shown in Equation (1) are shown by arrows in Fig. 6. TIFF0007649145000019.tif57 and the coordinates of the virtual imaging pixel shown in equation (20) Error with TIFF0007649145000020.tif612 g,h is expressed by the following equation (21).

number

[0043] error e g,h is the component e along the i direction of the captured image. 1,g,h and the component e along the j direction of the captured image. 2,g,h The correction unit 403 corrects the error e g,h and the error e g,h The coordinates of the corresponding measured captured pixels are stored in the storage unit 5 in association with each other.

[0044] [Creating error images] The correction unit 403 also estimates errors by interpolation processing for imaging pixels that are not associated with errors in the storage unit 5. First, the correction unit 403 acquires, from the storage unit 5, a plurality of errors of a plurality of measured imaging pixels that are associated with coordinates different from the coordinates of an imaging pixel that is not associated with an error. For example, the correction unit 403 acquires a plurality of errors that are associated in the storage unit 5 with coordinates that are within a predetermined distance from an imaging pixel that is not associated with an error.

[0045] The correction unit 403 estimates errors in the coordinates of imaging pixels that are not associated with errors by performing an interpolation process such as a linear interpolation process based on the acquired errors. The correction unit 403 creates an error image that associates the estimated errors with the coordinates of imaging pixels that correspond to the errors. The correction unit 403 includes in the error image a combination that associates the coordinates of the measured imaging pixels with the errors corresponding to the coordinates of the measured imaging pixels.

[0046] The error image is E g = (E 1,g , E 2,g )...(22) In equation (22), E 1,g indicates the error component along the horizontal direction of the coordinates of the measured image pixel, and E 2,g indicates an error component along the vertical direction of the coordinates of a measured captured pixel. When a plurality of measured captured images are generated by the measured coordinate specifying unit 401, the correction unit 403 creates a plurality of error images corresponding to the plurality of measured captured images. The correction unit 403 stores the created error images in the storage unit 5 as correction information for correcting the coordinates of the captured pixels. The error images can include errors that are difficult to express by the function of equation (16). For this reason, the correction unit 403 can improve the accuracy of image distortion correction by correcting the measurement result of the measurement object by referring to the function of equation (16) and the error images.

[0047] [Generating a single error image] The synthesis unit 404 acquires from the storage unit 5 an error associated with an imaging pixel in one error image and an error associated with the same imaging pixel in another error image. The synthesis unit 404 calculates a new error associated with this imaging pixel by averaging the acquired errors. The synthesis unit 404 generates a single error image by repeating the same process for other imaging pixels. For example, the synthesis unit 404 generates a single error image by averaging corresponding error components of the multiple error images as shown in the following equation (23).

number

[0048] The synthesis unit 404 is not limited to the example of averaging a plurality of error images to generate a single error image. For example, the synthesis unit 404 may average error images having relatively close coordinates in the depth direction to generate a plurality of error images. For example, the synthesis unit 404 may average error images No. 1 to No. 3 among the plurality of error images No. 1 to No. 8 to generate an error image, and average error images No. 2 to No. 4 to generate another error image, thereby generating a plurality of averaged error images. This may reduce the number of error images stored in the storage unit 5 to reduce the amount of processing and memory capacity, or may increase the number of error images to increase the correction accuracy. Note that the relative position and orientation information between the imaging device and the reference device obtained may be used as the coordinates in the depth direction of the reference device. Information on the installation position of the reference device specified in advance, or the result of measuring the three-dimensional coordinates of the feature points of the reference device using a combination of multiple imaging devices or an imaging device and a projection device, may also be used.

[0049] [Correction using geometric characteristics information] The shape specifying unit 405 causes the projection device 3 to project a projection image onto the measurement surface of the measurement object. The shape specifying unit 405 controls the first imaging device 1 and the second imaging device 2 to generate a measurement capture image by capturing an image of the measurement object with the projection image projected. Hereinafter, the measurement capture image generated by the first imaging device 1 may be referred to as the first measurement capture image, and the measurement capture image generated by the second imaging device 2 may be referred to as the second measurement capture image.

[0050] The shape specifying unit 405 corrects the coordinates of a plurality of measurement imaging pixels included in the measurement imaging image, based on a function included in the geometric characteristic information representing the geometric characteristic of the first imaging device 1 or the second imaging device 2 shown in equation (16) stored in the storage unit 5, and an error image serving as correction information created by the correction unit 403. The shape specifying unit 405 corrects the overall distortion for the coordinates of each measurement imaging pixel included in the measurement imaging image, using a function included in the geometric characteristic information of the first imaging device 1 or the second imaging device 2 shown in equation (16).

[0051] [Correction using a single error image] The shape specifying unit 405 reads out the single error image synthesized by the synthesizing unit 404 from the storage unit 5. The shape specifying unit 405 specifies the coordinates of a pixel in the error image that corresponds to the coordinates of the measurement imaging pixel. For example, the shape specifying unit 405 specifies the coordinates of a pixel in the error image that is closest to the coordinates of the measurement imaging pixel. The shape specifying unit 405 specifies an error associated with the coordinates of the pixel specified in the error image.

[0052] The shape specifying unit 405 corrects the coordinates of the measurement imaging pixel based on the specified error. For example, the shape specifying unit 405 corrects the coordinates (i, j) of the measurement imaging pixel included in the measurement imaging image by subtracting an error component E 1,g Shift the coordinates horizontally by (i,j) and calculate the error component E 2,g Correct the local distortion by shifting the coordinate vertically by (i,j).

[0053] Specify Projected Coordinates The shape specifying unit 405 calculates three-dimensional measurement coordinates of the measurement point of the measurement object based on the coordinates of the measurement image-captured pixel after correction. In this embodiment, an example will be described in which the shape specifying unit 405 calculates the three-dimensional measurement coordinates of the measurement point by specifying a correspondence relationship between the first measurement image-captured pixel of the first measurement image-captured image and the second image-captured pixel of the second measurement image-captured image.

[0054] Fig. 7 is a diagram showing examples of types of projection images projected by the shape specifying unit 405. The black areas in Fig. 7 indicate non-projection areas onto which the shape specifying unit 405 does not project light, and the white areas indicate light projection areas onto which the shape specifying unit 405 projects light.

[0055] Fig. 7(a) is a reference pattern (all black pattern) in which no light is projected onto the entire object to be measured. Fig. 7(b) is a reference pattern (all white pattern) in which light is projected onto the entire object to be measured. Figs. 7(c) to 7(f) are made up of light projection areas and non-projection areas, and show binary stripe patterns in which stripes of different widths are arranged in the same direction for each projected image. Details will be described later, but the stripe pattern shown in Fig. 7 corresponds to a gray code and is used to identify projection coordinates that indicate the position of the projected pixel of the projected image that corresponds to the measurement capture pixel of the measurement capture image.

[0056] The shape specifying unit 405 projects a projection image including a grayscale stripe pattern having a sinusoidal luminance distribution onto the measurement target. FIGS. 8(a) to (d) show examples of grayscale stripe patterns having a sinusoidal luminance distribution. The binary stripe patterns of FIGS. 7(c) to (f) are binary images consisting of black and white regions, whereas the grayscale stripe patterns of FIGS. 8(a) to (d) have a sine wave-like change in shading from the white region to the black region along the stripe width direction. The grayscale stripe patterns of FIGS. 8(a) to (d) have a constant stripe spacing, and the spatial frequency of the stripes of these grayscale stripe patterns is, for example, four times that of the binary stripe pattern of FIG. 7(f).

[0057] The gradation stripe patterns in Fig. 8(a) to (d) show the same luminance distribution, except that the phases of the sine waves showing the luminance distribution are different by 90 degrees. In this embodiment, the shape specifying unit 405 projects a total of 10 projection images, including two reference patterns shown in Fig. 7(a) and (b), four binary stripe patterns shown in Fig. 7(c) to (f), and four gradation stripe patterns shown in Fig. 8(a) to (d). The gradation stripe pattern shown in Fig. 8 is used to specify the projection coordinates together with the stripe pattern shown in Fig. 7. The shape specifying unit 405 may also project a plurality of projection images including a pattern in which the width of the stripe varies depending on the direction in which the stripe extends, onto the measurement object.

[0058] As described above, the binary stripe patterns shown in Figures 7(c) to 7(f) correspond to Gray codes. Figure 9 shows an example of a Gray code corresponding to the binary stripe patterns shown in Figures 7(c) to 7(f). By associating 0 in the Gray code with the non-projection area and 1 with the light projection area, the binary stripe patterns shown in Figures 7(c) to 7(f) are generated.

[0059] 7 and 9, each position in the x-direction is represented by a code value that combines the numbers 0 or 1 at the corresponding positions of each Gray code. Position 0 in Fig. 9 corresponds to the code value "0000", position 1 corresponds to the code value "0001", and position 15 corresponds to the code value "1000".

[0060] The shape specifying unit 405 generates measurement captured images by capturing the projected images projected onto the measurement object by the first imaging device 1 and the second imaging device 2. The shape specifying unit 405 specifies projection coordinates indicating the positions of projected pixels corresponding to the measurement captured pixels included in the measurement captured image based on the pattern included in the measurement captured image. The shape specifying unit 405 specifies projection coordinates corresponding to the measurement captured pixels included in the measurement captured image by analyzing changes in shading in the pattern included in the measurement captured image.

[0061] The shape specifying unit 405 calculates the average of the luminance value when the all black pattern shown in FIG. 7(a) is projected and the luminance value when the all white pattern shown in FIG. 7(b) is projected for each pixel as the intermediate value. Similarly, the shape specifying unit 405 compares the luminance value of each measurement image pixel in the four measurement images in a state where the binary stripe patterns shown in FIG. 7(c) to FIG. 7(f) are projected onto the measurement target with the corresponding intermediate value to specify the code value of each measurement image pixel. By specifying the code value, the shape specifying unit 405 can specify the position toward which the projected binary stripe is reflected at the position of each measurement image pixel. The shape specifying unit 405 specifies the position from position 0 to position 15 shown in FIG. 9 in which each measurement image pixel included in the measurement image is included.

[0062] Furthermore, the shape specifying unit 405 specifies, as projection coordinates, the phase of a sine wave corresponding to a measurement capture pixel in a measurement capture image when a gradation stripe pattern having a sine wave-like luminance distribution is projected onto the measurement target. Since the gradation stripe pattern of the projection image has periodicity, multiple projection pixels have the same projection coordinates in the projection image. Hereinafter, the projection coordinates having periodicity in the projection image are also referred to as relative projection coordinates. On the other hand, the projection coordinates that are uniquely determined in the projection image are also particularly referred to as absolute projection coordinates.

[0063] FIG. 10 is a diagram showing the relationship between absolute projection coordinates and relative projection coordinates. The vertical axis of FIG. 10 indicates projection coordinates. The horizontal axis of FIG. 10 indicates the position of a projection pixel in the width direction of a stripe included in a projection image. The width direction is a direction perpendicular to the direction in which the stripe extends. As shown by the solid line in FIG. 10, the relative projection coordinates have periodicity. The relative projection coordinates indicate the same value for each period of the repetition of a gradation stripe pattern having a sinusoidal luminance distribution. On the other hand, as shown by the dashed line in FIG. 10, the absolute projection coordinates are uniquely determined in a projection image.

[0064] The shape specifying unit 405 specifies relative projection coordinates corresponding to the measurement imaging pixel by analyzing the shading of the grayscale stripe pattern. The shape specifying unit 405 specifies which position, from position 0 to position 15, the measurement imaging pixel corresponds to based on the gray code indicated by the binary stripe pattern, and thereby specifies absolute projection coordinates corresponding to the measurement imaging pixel.

[0065] [Identifying the correspondence between 3D measured coordinates] The correspondence between the first measurement imaging pixel included in the first measurement image generated by the first imaging device 1 and the second measurement imaging pixel included in the second measurement image generated by the second imaging device 2 can be found using an epipolar line. Fig. 11 is a diagram showing the correspondence between the first measurement imaging pixel A and the second measurement imaging pixel B. The left side of Fig. 11 shows the image plane of the first measurement imaging image, and the right side of Fig. 11 shows the image plane of the second measurement imaging image.

[0066] On the image plane of the second measurement image, an epipolar line E corresponding to the first measurement image pixel A is BA If the first measurement imaging pixel A and the second measurement imaging pixel B correspond to the same measurement point M of the measurement object, the second measurement imaging pixel B corresponds to the epipolar line E BA It's on top.

[0067] The shape specifying section 405 specifies a second measurement imaging pixel B that corresponds to the same measurement point M as the first measurement imaging pixel A on the measurement object. Specifically, the shape specifying section 405 specifies an epipolar line E BA A second measurement imaging pixel B is identified that is on the first measurement imaging pixel A and has the same absolute projection coordinates as the first measurement imaging pixel A. If the subscript indicating the first imaging device 1 is α, then equation (19) can be rewritten as the following equation (24).

number

[0068] If the subscript indicating the second imaging device 2 is β, then equation (19) can be rewritten as equation (25) below.

number

[0069] The shape specifying unit 405 calculates the three-dimensional measurement coordinates of the other measurement points of the measurement object in a similar manner. The shape specifying unit 405 specifies the shape of the measurement object based on the calculated three-dimensional coordinates of the multiple measurement points. In addition, the shape specifying unit 405 is not limited to the example of calculating the three-dimensional measurement coordinates of the measurement point by specifying the correspondence between the first measurement imaging pixel A and the second imaging pixel B. For example, the shape specifying unit 405 may calculate the three-dimensional measurement coordinates of the measurement point by specifying a combination of the first measurement imaging pixel A and a projection pixel included in the projection image.

[0070] [Variation for selecting an error image to be used for correction] The shape specifying unit 405 may select an error image to be used for correction for each measurement imaging pixel. The correcting unit 403 generates an error table in which the coordinates in the depth direction where the reference tool 51 is installed are associated with the error image, and stores the error table in the storage unit 5.

[0071] The shape specifying unit 405 obtains approximate values ​​of the measurement three-dimensional coordinates corresponding to the measurement captured pixels included in the measurement captured image. For example, the shape specifying unit 405 obtains approximate values ​​of the measurement three-dimensional coordinates of the measurement point of the measurement object without using an error image. The shape specifying unit 405 may obtain approximate values ​​of the measurement three-dimensional coordinates by reducing the number of significant digits of the data. The shape specifying unit 405 obtains the error table from the storage unit 5.

[0072] The shape identification unit 405 refers to an error table including a plurality of error images, and compares the depth coordinate of the position of the reference tool corresponding to each error image with the depth coordinate of the obtained approximate value. The shape identification unit 405 selects an error image based on the comparison result. For example, the shape identification unit 405 selects an error image associated with the depth coordinate closest to the depth coordinate of the obtained approximate value.

[0073] The shape specifying unit 405 calculates the measurement three-dimensional coordinates by referring to the selected error image. For example, the shape specifying unit 405 specifies an error associated with the coordinates of the imaging pixel that is closest to the approximate value of the measurement three-dimensional coordinates. The shape specifying unit 405 corrects the coordinates of the measurement imaging pixel based on the specified error. The shape specifying unit 405 calculates the measurement three-dimensional coordinates of the measurement point based on the coordinates of the measurement imaging pixel after the correction.

[0074] Furthermore, the shape specifying unit 405 may select multiple error images close to the depth coordinate of the approximate value, and obtain the value of the error image corresponding to the depth coordinate of the approximate value by interpolation. As an example, the shape specifying unit 405 may use the selected multiple error images to obtain the value of the error image corresponding to the depth coordinate of the approximate value by linear interpolation. This allows for more accurate correction.

[0075] The following method may be used to obtain the approximate value. For example, in the measurement system of FIG. 1, measurements can be performed using a combination of multiple devices, such as a three-dimensional coordinate calculation using a combination of an imaging device and a projection device, or a three-dimensional coordinate calculation using two imaging devices and a projection device. The result of a certain measurement system among the multiple measurement systems obtained in this way may be used as an approximate value. For example, the three-dimensional coordinate of a combination of an imaging device and a projection device can be used as an approximate value for measurement using two imaging devices and a projection device. In this way, the measurement efficiency can be improved.

[0076] The shape specifying unit 405 is not limited to the example of correcting image distortion using the error image. For example, the shape specifying unit 405 may correct image distortion of the projection device 3 using the error image. The projection device 3 is similar to the first imaging device 1 except that the light travels in the opposite direction. Therefore, the shape specifying unit 405 can correct image distortion of the projection device 3 in the same manner as the first imaging device 1.

[0077] [Processing procedure for generating error images] 12 is a flowchart showing a processing procedure for generating an error image by the three-dimensional shape measuring device 100. This processing procedure starts, for example, when an operation receiving unit (not shown) receives a user operation to instruct calibration of a function included in the geometric characteristic information of the first imaging device 1 stored in the storage unit 5 with the reference tool 51 installed.

[0078] First, the actual coordinate identifying unit 401 generates an actual captured image by having the first imaging device 1 capture an image of a reference instrument 51 having a plurality of feature points. The actual coordinate identifying unit 401 identifies coordinates of actual captured pixels corresponding to feature points of the reference instrument 51 by analyzing the generated actual captured image (S101). The virtual coordinate identifying unit 402 reads out geometric characteristic information including parameters and functions representing geometric characteristics of the first imaging device 1 and position and orientation information identifying the position and orientation of the reference instrument 51 from the storage unit 5. The virtual coordinate identifying unit 402 identifies coordinates of virtual captured pixels using the read geometric characteristic information and position and orientation information (S102).

[0079] The correction unit 403 identifies an error occurring between the coordinates of the virtual captured pixel and the coordinates of the measured captured pixel (S103). The measured coordinate identifying unit 401 judges whether errors corresponding to all measured captured pixels have been identified (S104). When it is judged in the judgment of S104 that errors corresponding to all measured captured pixels have been identified (YES in S104), the measured coordinate identifying unit 401 performs an interpolation process based on the acquired multiple errors (S105) and estimates an error corresponding to an captured pixel with no associated error.

[0080] The correction unit 403 creates an error image by associating the estimated error with the coordinates of the captured pixels corresponding to the error (S106), and ends the process. If the actual coordinate specification unit 401 determines in the determination of S104 that errors corresponding to all the actually measured captured pixels have not been specified (NO in S104), the process returns to S101.

[0081] [Processing procedure for identifying the shape of the measurement target] 13 is a flowchart showing a processing procedure for specifying the shape of a measurement object by three-dimensional shape measuring device 100. This processing procedure starts when the operation receiving unit receives a user operation instructing to specify the shape of the measurement object.

[0082] First, the shape specifying unit 405 generates measurement captured images by capturing images projected onto the measurement object by the first imaging device 1 and the second imaging device 2 (S201). The shape specifying unit 405 corrects the coordinates of each measurement captured pixel included in the measurement captured image by using a function indicating the geometric characteristics of the first imaging device 1, etc. The shape specifying unit 405 corrects the coordinates of each measurement captured pixel included in the measurement captured image by referring to the single error image synthesized by the synthesis unit 404 (S202).

[0083] The shape specifying unit 405 analyzes the change in shading in the pattern included in the corrected measurement captured image to specify the projection coordinates indicating the position of the projection pixel corresponding to the measurement captured pixel (S203).The shape specifying unit 405 calculates the measurement 3D coordinates of the measurement point of the measurement object by specifying a combination of the first measurement captured pixel and the second measurement captured pixel corresponding to the same measurement point of the measurement object (S204).

[0084] The shape specifying unit 405 determines whether there is another measurement imaging pixel that has not been used in the calculation of the measurement three-dimensional coordinates (S205). If there is no other measurement imaging pixel that has not been used in the calculation of the measurement three-dimensional coordinates in the determination of S205 (NO in S205), the shape specifying unit 405 specifies the shape of the measurement target based on the measurement three-dimensional coordinates of the multiple measurement points (S206). If there is another measurement imaging pixel that has not been used in the calculation of the measurement three-dimensional coordinates in S205 (YES in S205), the shape specifying unit 405 returns to the process of S202.

[0085] [Effects of the present invention] In the three-dimensional shape measuring apparatus 100 of this embodiment, the shape specifying section 405 corrects the coordinates of a plurality of measurement imaging pixels included in the measurement imaging image based on a function indicating the geometric characteristics of the first imaging device 1 and correction information for correcting the coordinates of the measurement imaging pixels. Therefore, the shape specifying section 405 can highly accurately correct image distortion that is difficult to express by a function indicating the geometric characteristics.

[0086] [Modification for imaging a reference instrument rotated around the depth axis] The measured coordinate identifying unit 401 may generate a plurality of measured captured images by gradually rotating the reference instrument 51 around an axis in the depth direction. The correcting unit 403 generates error images corresponding to these measured captured images. The synthesizing unit 404 may generate an error image by averaging the generated plurality of error images. In this way, the synthesizing unit 404 can average the variation in the coordinates in the depth direction of the feature points of the reference instrument 51, thereby further improving the measurement accuracy.

[0087] [Modification for calculating the correction function] The correction unit 403 may use a plurality of error images to create a correction function for correction for each imaging pixel. Fig. 14 is a diagram showing the relationship between the depth coordinate and the horizontal component of the error in the coordinate of a specific measured imaging pixel. The horizontal axis of Fig. 14 indicates the depth coordinate at which the reference tool 51 is installed. The vertical axis of Fig. 14 indicates the horizontal component of the error in the coordinate of a specific imaging pixel. The multiple black circles in Fig. 14 indicate the distribution of the horizontal component of the error in the coordinate of a specific imaging pixel.

[0088] The horizontal component of the error in the coordinates of the imaging pixels varies depending on the coordinate in the depth direction. The correction unit 403 may generate a correction function that matches the distribution of the black dots of the horizontal component of the error in the coordinates of the imaging pixels, as shown in the curve in FIG. 14. For example, the correction unit 403 generates a correction function that matches the distribution of the black dots in FIG. 14 using a known method such as the least squares method. The correction unit 403 associates the generated correction function with the vertical coordinates of the imaging pixels and the horizontal coordinates of the imaging pixels, and stores them in the storage unit 5. The correction unit 403 also creates a similar correction function for the vertical component of the error in the coordinates of the imaging pixels.

[0089] The shape specifying unit 405 corrects the coordinates of the measurement imaging pixel by referring to the correction function generated by the correction unit 403, instead of the error image. By referring to the correction function, the shape specifying unit 405 can obtain the error in the coordinates of the measurement imaging pixel corresponding to any coordinate in the depth direction, and therefore can correct the distortion of the image with higher accuracy.

[0090] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0091] 1 First imaging device 2 Second imaging device 3 Projection device 4. Control section 5 Storage section 11 Optical elements 12 Image sensor 21 Optical elements 22 Image sensor 51 Reference Instrument 100 Three-dimensional shape measuring device 401 Actual coordinate identification unit 402 Virtual coordinate identification unit 403 Correction Section 404 Synthesis section 405 Shape identification part

Claims

1. A three-dimensional shape measuring apparatus including an imaging device, the three-dimensional shape measuring apparatus measuring a three-dimensional shape of a measurement object based on a measurement captured image obtained by capturing an image of the measurement object by the imaging device, a measured coordinate specifying unit that specifies coordinates of measured captured pixels corresponding to a plurality of feature points in a measured captured image generated by capturing an image of a reference tool having the plurality of feature points by the imaging device; a virtual coordinate specification unit that virtually projects feature points on the reference instrument onto an image plane of the imaging device using geometric characteristic information that represents geometric characteristics of the imaging device and relative position and orientation information between the imaging device and the reference instrument, and specifies coordinates of virtual imaging pixels that correspond to the feature points on the image plane; a correction unit that creates an error image as correction information for correcting the coordinates of the measurement imaging pixel included in the measurement imaging image based on a comparison between the coordinates of the measured imaging pixel and the coordinates of the virtual imaging pixel, the correction unit identifying an error occurring between the coordinates of the virtual imaging pixel and the coordinates of the measured imaging pixel, and creating a plurality of the error images that associate the identified error with the coordinates of the pixel corresponding to the error, based on a plurality of measured imaging images that are captured by rotating the reference instrument around an axis in the depth direction, the plurality of measured imaging images being generated by capturing an image of the reference instrument installed in a plurality of positions and orientations using the imaging device; a combiner for generating one or more averaged error images by averaging an error associated with the measurement imaging pixel in one of the error images with an error associated with the same measurement imaging pixel in another of the error images; a shape specifying unit that corrects coordinates of a plurality of measurement imaging pixels included in a measurement imaging image generated by the imaging device by imaging the measurement object based on the one or more error images, and specifies a shape of the measurement object based on measurement three-dimensional coordinates calculated based on the corrected pixel coordinates; A three-dimensional shape measuring apparatus comprising:

2. the correction unit estimates an error of the pixel to which the error is not associated based on a plurality of errors of a plurality of the actually measured captured pixels associated with coordinates different from coordinates of the pixel to which the error is not associated, and generates the error image by associating the error with the coordinates of the pixel corresponding to the error. The three-dimensional shape measuring apparatus according to claim 1 .

3. the shape specifying unit obtains approximate values ​​of measurement three-dimensional coordinates corresponding to the measurement imaging pixels included in the measurement imaging image, and calculates the measurement three-dimensional coordinates based on errors specified using a plurality of error images, a depth direction coordinate of the position of the reference instrument corresponding to the error images, and a depth direction coordinate of the approximate values. The three-dimensional shape measuring apparatus according to claim 1 or 2.

4. A projection device for projecting a projection image is further provided, The imaging device generates the measurement captured image by capturing an image of the measurement object onto which the projection image is projected. The three-dimensional shape measuring apparatus according to claim 1 .

5. A three-dimensional shape measuring method for measuring a three-dimensional shape of a measurement object based on a measurement captured image obtained by capturing an image of the measurement object by an imaging device, comprising: A step of identifying coordinates of measured captured pixels corresponding to a plurality of feature points in a measured captured image generated by capturing an image of a reference tool having the plurality of feature points by the imaging device; a step of virtually projecting feature points on the reference instrument onto an image plane of the imaging device using geometric characteristic information representing geometric characteristics of the imaging device and relative position and orientation information between the imaging device and the reference instrument, and specifying coordinates of virtual imaging pixels corresponding to the feature points on the image plane; a step of creating an error image as correction information for correcting the coordinates of the measurement imaging pixel included in the measurement imaging image based on a comparison between the coordinates of the measured imaging pixel and the coordinates of the virtual imaging pixel, the step of identifying an error occurring between the coordinates of the virtual imaging pixel and the coordinates of the measured imaging pixel, and creating a plurality of the error images in which the identified error is associated with the coordinates of the pixel corresponding to the error, the error images being based on a plurality of measured imaging images captured by rotating the reference instrument around an axis in the depth direction, the measured imaging images being generated by capturing an image of the reference instrument installed in a plurality of positions and orientations using the imaging device; generating one or more averaged error images by averaging an error associated with the measurement imaging pixel in one of the error images with an error associated with the same measurement imaging pixel in another of the error images; a step of correcting coordinates of a plurality of measurement imaging pixels included in a measurement imaging image generated by the imaging device by imaging the measurement object based on the one or more error images, and specifying a shape of the measurement object based on measurement three-dimensional coordinates calculated based on the corrected pixel coordinates; A three-dimensional shape measuring method comprising the steps of:

Images