3D Measurement System, Apparatus, Method, and Program

Abstract

Provided is a three-dimensional measurement system comprising: a projector for projecting, onto an object, striped pattern light with periodically repeating light and dark parts in a first direction; a camera for photographing the pattern light and generating a photography image; a corrected camera coordinate acquisition unit for acquiring corrected camera coordinates that correspond to a pixel of interest and are for the first direction and a second direction orthogonal to the first direction; a decoding processing unit for acquiring decoding information relating to the first direction; a corresponding projector coordinate acquisition unit for acquiring a corresponding projector coordinate that is on a virtual logical image observed by a virtual camera disposed at the projector position, corresponds to the pixel of interest, and is for the first direction; a corrected projector coordinate acquisition unit for acquiring a corrected projector coordinate that is for the first direction and corresponds to the result of subjecting the corresponding projector coordinate to distortion correction and parallelization conversion processing relating to the projector; and a distance information generation unit for generating information about the distance from the camera or projector to the object.

Description

Technical Field

[0001] This invention relates to a three-dimensional measurement system and the like.

Background Art

[0002] As a method for non-contact acquisition of three-dimensional information such as depth information, an active stereo method is known.

[0003] In the active stereo method, a striped pattern light in which bright and dark parts are periodically repeated is projected from a projector onto an object, and the projected pattern light is photographed by a camera. By analyzing the photographed image, the distance from the projector or the camera to the object can be calculated.

[0004] By the way, in the optical system of a projector including a lens or the like, thermal expansion and thermal contraction occur according to the temperature. When such thermal expansion or the like occurs, distortion occurs in the pattern light and an analysis error occurs in the image, which may lead to a decrease in the accuracy of three-dimensional measurement. In order to prevent such inconveniences, various countermeasures have been taken conventionally.

[0005] For example, a method of performing distortion correction of the optical system related to a projector during image analysis has been known (for example, Non-Patent Document 1). According to such a method, for example, by correcting using a calibration parameter obtained by previously measuring the distortion caused by thermal expansion of the optical system related to the projector, a decrease in three-dimensional measurement accuracy can be prevented.

[0006] Also, although it is an example of a camera, a method of adopting an optical system with a temperature compensation function for the optical system and suppressing the distortion itself generated in the optical system has also been known (for example, Patent Document 1). According to such a method, the distortion of the optical system due to heat is suppressed by a physical / mechanical method. By diverting such a method, a decrease in three-dimensional measurement accuracy can be prevented.

Prior Art Documents

Patent Documents

[0007] [Non-Patent Document 1] Hiroshi Kawasaki, Hiroshi Ohsawa, Ryo Furukawa, Yasuaki Nakamura, "Dense 3D Reconstruction with an Uncalibrated Stereo System using Coded Structured Light", 2005 IEEE Computer Science Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, 2005, pp.107-107, doi: 10.1109 / CVPR.2005.440

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 05-303032 [Summary of the Invention] [Problems to be Solved by the Invention]

[0009] However, in a method of performing distortion correction of the optical system related to the projector during image analysis, in three-dimensional measurement, it was necessary to project horizontal stripe pattern light and vertical stripe pattern light. Therefore, it took a certain amount of time to project the pattern light, and as a result, it was difficult to perform high-speed three-dimensional measurement.

[0010] Also, in a method of adopting an optical system with a temperature compensation function, the optical system related to the projector becomes sophisticated and the cost increases, which may lead to an increase in the cost of the entire three-dimensional measurement system.

[0011] The present invention has been made in view of the above technical background, and an object thereof is to realize high-speed three-dimensional measurement while correcting distortion occurring in the optical system of the projector by image processing. [Means for Solving the Problems]

[0012] The above technical problem can be solved by a three-dimensional measurement system or the like having the following configuration.

[0013] That is, the three-dimensional measurement system according to the present invention includes a projector that projects striped pattern light in which light and dark portions are periodically repeated with respect to a first direction onto an object, a camera that captures the pattern light projected onto the object to generate a captured image, and a correction camera coordinate acquisition unit that performs distortion correction processing and parallelization conversion processing related to the camera on a target pixel on the captured image to acquire correction camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction, a decoding processing unit that performs decoding processing related to the pattern light on the captured image to acquire decoding information related to the first direction, a corresponding projector coordinate acquisition unit that is on a virtual logical image observed in a virtual camera disposed at the projector position, corresponds to the target pixel, and acquires corresponding projector coordinates related to the first direction based on the decoding information, a correction projector coordinate acquisition unit that acquires correction projector coordinates related to the first direction corresponding to a result of performing distortion correction and parallelization conversion processing related to the projector on the corresponding projector coordinates based on the correction camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction, and a distance information generation unit that generates distance information from the camera or the projector to the object based on the correction camera coordinates related to the first direction and the correction projector coordinates related to the first direction.

[0014] According to such a configuration, by simply projecting pattern light in one direction, correction considering distortion generated in the optical system related to the projector can be performed. Thereby, while correcting the distortion generated in the optical system of the projector by image processing, high-speed three-dimensional measurement can be realized.

[0015] The first direction may be substantially parallel to an imaginary line connecting the projector and the camera.

[0016] According to such a configuration, by performing a parallelization transformation, distance information can be generated by utilizing the fact that the corrected camera coordinates in the second direction and the corrected projector coordinates in the second direction coincide.

[0017] In the corrected projector coordinate acquisition unit, the corrected projector coordinates in the first direction may be stored in a two-dimensional look-up table in which the corrected camera coordinates in the second direction are in a first series and the corresponding projector coordinates in the first direction are in a second series.

[0018] According to such a configuration, the amount of calculation can be reduced, and further, high-speed three-dimensional measurement can be realized.

[0019] The two-dimensional look-up table may be generated based on the result of projecting a striped pattern light in which bright and dark portions are periodically repeated in the first direction and a striped pattern light in which bright and dark portions are periodically repeated in the second direction onto a predetermined object and photographing the projected pattern light.

[0020] According to such a configuration, even when only the pattern light in one direction is projected during three-dimensional measurement, appropriate distance information can be generated.

[0021] A temperature sensor that directly or indirectly measures the temperature of the optical system related to the projector, a storage unit that stores a plurality of two-dimensional look-up tables for each temperature, which are generated for each temperature and have the corrected camera coordinates in the second direction in a first series and the corresponding projector coordinates in the first direction in a second series, a selective reading unit that reads out the corresponding two-dimensional look-up table for each temperature according to the temperature measured by the temperature sensor, and the projector coordinate acquisition unit may obtain the corrected projector coordinates in the first direction by using the read two-dimensional look-up table for each temperature instead of the two-dimensional look-up table.

[0022] According to such a configuration, it is possible to obtain projector coordinates for which appropriate correction processing is performed according to the temperature of the optical system related to the projector.

[0023] The corrected projector coordinates in the first direction may be generated by performing interpolation processing using the values stored in the two-dimensional look-up table for each temperature corresponding to the temperatures before and after the temperature measured by the temperature sensor.

[0024] According to such a configuration, by preparing two-dimensional look-up tables for each temperature at appropriate intervals, it is possible to obtain appropriate corrected projector coordinates at any temperature.

[0025] The interpolation processing may be linear interpolation processing.

[0026] According to such a configuration, it is possible to obtain appropriate corrected projector coordinates at any temperature by linear interpolation.

[0027] The pattern light may be pattern light related to a spatial coding method.

[0028] According to such a configuration, it is possible to realize a spatial coding method.

[0029] The pattern light may be pattern light related to a phase shift method.

[0030] According to such a configuration, the phase shift method can be realized.

[0031] Viewed from another aspect, the present invention is a three-dimensional measurement apparatus, including: a projector that projects striped pattern light in which bright and dark portions are periodically repeated in a first direction onto an object; a camera that photographs the pattern light projected onto the object to generate a photographed image; a corrected camera coordinate acquisition unit that, for a target pixel on the photographed image, performs distortion correction processing and parallelization conversion processing related to the camera to acquire corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding processing unit that performs decoding processing related to the pattern light on the photographed image to acquire decoding information related to the first direction; a corresponding projector coordinate acquisition unit that, based on the decoding information, acquires corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the projector position; a corrected projector coordinate acquisition unit that, based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction, acquires corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction and parallelization conversion processing related to the projector on the corresponding projector coordinates; and a distance information generation unit that generates distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

[0032] Viewed from another aspect, the present invention is a method for controlling a three-dimensional measurement system, comprising: a projector that projects striped pattern light in which bright and dark portions are periodically repeated in a first direction onto an object; and a camera that captures the pattern light projected onto the object to generate a captured image. The method for controlling a three-dimensional measurement system includes: a corrected camera coordinate acquisition step of performing a distortion correction process and a parallelization conversion process related to the camera on a target pixel on the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding process step of performing a decoding process related to the pattern light on the captured image to obtain decoding information related to the first direction; a corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates related to the first direction that are on a virtual logical image observed in a virtual camera disposed at the projector position and correspond to the target pixel based on the decoding information; a corrected projector coordinate acquisition step of obtaining corrected projector coordinates related to the first direction that correspond to a result of performing a distortion correction and a parallelization conversion process related to the projector on the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction; and a distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

[0033] Viewed from another aspect, the present invention is a control program for a three-dimensional measurement system, comprising: a projector that projects striped pattern light in which bright and dark portions are periodically repeated with respect to a first direction onto an object; and a camera that captures the pattern light projected onto the object to generate a captured image. The control program for the three-dimensional measurement system includes: a corrected camera coordinate acquisition step of performing distortion correction processing and parallelization conversion processing related to the camera on a target pixel on the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding processing step of performing decoding processing related to the pattern light on the captured image to obtain decoding information related to the first direction; a corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the projector position based on the decoding information; a corrected projector coordinate acquisition step of obtaining corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction and parallelization conversion processing related to the projector on the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction; and a distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

[0034] Viewed from another aspect, the present invention is a three-dimensional camera, comprising: a projector that projects striped pattern light in which bright and dark portions are periodically repeated with respect to an object in a first direction; a camera that captures the pattern light projected onto the object to generate a captured image; a corrected camera coordinate acquisition unit that performs a distortion correction process and a parallelization conversion process related to the camera on a target pixel in the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding process unit that performs a decoding process related to the pattern light on the captured image to obtain decoding information related to the first direction; a corresponding projector coordinate acquisition unit that, based on the decoding information, obtains corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the projector position; a corrected projector coordinate acquisition unit that, based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction, obtains corrected projector coordinates related to the first direction corresponding to a result of performing a distortion correction and a parallelization conversion process related to the projector on the corresponding projector coordinates; and a distance information generation unit that generates distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

[0035] Viewed from another aspect, the present invention is a three-dimensional measurement system, comprising: a projector that projects striped pattern light in which light and dark portions are periodically repeated with respect to a first direction onto an object; a camera that captures the pattern light projected onto the object to generate a captured image; a corrected camera coordinate acquisition unit that performs distortion correction processing related to the camera on a target pixel in the captured image to acquire corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding processing unit that performs decoding processing related to the pattern light on the captured image to acquire decoding information related to the first direction; a corresponding projector coordinate acquisition unit that, based on the decoding information, acquires corresponding projector coordinates that are on a virtual logical image observed in a virtual camera disposed at the projector position, correspond to the target pixel, and are related to the first direction; a corrected projector coordinate acquisition unit that, based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction, acquires corrected projector coordinates related to the first direction that correspond to a result of performing distortion correction processing related to the projector on the corresponding projector coordinates; and a distance information generation unit that generates distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

[0036] According to such a configuration, by simply projecting pattern light in one direction, correction can be performed taking into account distortion generated in the optical system of the projector. As a result, high-speed three-dimensional measurement can be realized while correcting distortion occurring in the optical system of the projector by image processing.

[0037] Viewed from another aspect, the present invention is a method for controlling a three-dimensional measurement system, comprising: a projector that projects striped pattern light in which bright and dark portions are periodically repeated with respect to a first direction onto an object; and a camera that captures the pattern light projected onto the object to generate a captured image. The method for controlling a three-dimensional measurement system includes: a corrected camera coordinate acquisition step of performing distortion correction processing on a camera with respect to a target pixel on the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding processing step of performing decoding processing on the captured image with respect to the pattern light to obtain decoding information related to the first direction; a corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the projector position based on the decoding information; a corrected projector coordinate acquisition step of obtaining corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction processing on the projector with respect to the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction; and a distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

[0038] Viewed from another aspect, the present invention is a control program for a three-dimensional measurement system, which includes a projector that projects striped pattern light in which bright and dark portions are periodically repeated in a first direction onto an object, and a camera that captures the pattern light projected onto the object to generate a captured image. The control program for the three-dimensional measurement system includes: a corrected camera coordinate acquisition step of performing distortion correction processing related to the camera on a target pixel in the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; a decoding processing step of performing decoding processing related to the pattern light on the captured image to obtain decoding information related to the first direction; a corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the projector position based on the decoding information; a corrected projector coordinate acquisition step of obtaining corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction processing related to the projector on the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction; and a distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction.

Advantages of the Invention

[0039] According to the present invention, by simply projecting pattern light in one direction, correction considering distortion generated in the optical system of the projector can be performed. Thereby, while correcting the distortion generated in the optical system of the projector by image processing, high-speed three-dimensional measurement can be realized.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

[0041] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0042] (1. First Embodiment) As a first embodiment, an example in which the present invention is applied to a three-dimensional measurement system will be described. Here, the three-dimensional measurement system includes at least a device or system capable of generating three-dimensional information such as depth information, depth information, or distance information, and may be referred to by other terms such as a three-dimensional camera. Further, it may be configured as a device further provided with other functions.

[0043] (1.1 Configuration of the System) Figure 1 is an overall configuration diagram of a three-dimensional measurement system 100 according to the present embodiment. As is clear from the figure, the three-dimensional measurement system 100 includes a controller unit 1 and a sensor head unit 3 connected to the controller unit 1. Using these configurations, three-dimensional information of the object 5 is generated.

[0044] The sensor head unit 3 includes a camera (imaging device) 31 and a projector (projection device) 35 that is horizontally adjacent to the camera 31. Hereinafter, an axis parallel to the imaging surface of the camera 31 or the projection surface of the projector 35 may be referred to as the x-axis, and a direction perpendicular to the paper surface may be referred to as the y-axis.

[0045] The camera 31 includes an image sensor 311 and an objective lens 312, and is arranged so as to accommodate the object 5 within its angle of view. As will be described later, the image information generated by the image sensor 311 is taken into the controller unit 1 by the image acquisition unit 15.

[0046] The projector 35 includes a light source 351, and on its optical path, in order from the light source 351 side, it is provided with a plane wave conversion lens 352, a mirror 353, a DMD 355, a relay lens 356, and an objective lens 357. Note that DMD 355 is an abbreviation for Digital Micromirror Device, and it is a device that controls the direction of light in pixel units by rapidly tilting drive extremely small mirrors arranged in an array. The projector 35 is arranged at a position where it can project the pattern light described later onto the object 5. Further, it is configured to be able to project a desired pattern light under the control of the projection control unit 12 described later.

[0047] The controller unit 1 includes an input reception unit 11 that receives inputs such as a measurement start signal, a projection control unit 12 that controls the projection of pattern light in the projector 35, and a storage unit 13 that stores various information such as a two-dimensional look-up table described later. Further, the controller unit 1 further includes an image acquisition unit 15 that acquires an image captured by the camera 31, an image processing unit 16 that performs image processing on the acquired image, and an output unit 17 that outputs three-dimensional information. Note that these functions are realized by a control device or circuit such as a CPU that executes a program.

[0048] Note that the configurations of the camera 31 and the projector 35 are both exemplary and schematic configurations. Therefore, other configurations such as those including additional lenses may also be possible.

[0049] Also, in the present embodiment, although the controller unit 1 and the sensor head unit 3 are configured separately, the present invention is not limited to such a configuration. Therefore, all the configurations may be arranged in one device, or some of them may be configured as separate devices.

[0050] (1.2 Operation of the System) Next, with reference to FIGS. 2 to 17, the operation of the three-dimensional measurement system 100 will be described. In the three-dimensional measurement system 100 according to the present embodiment, predetermined preprocessing is performed before the actual three-dimensional measurement process. Each process will be described in order below.

[0051] (1.2.1 Preprocessing) FIG. 2 is a general flowchart of the preprocessing. As is apparent from the figure, when the process starts, the pair image generation process is executed (S11).

[0052] FIG. 3 is a detailed flowchart of the pair image generation process (S11). As is apparent from the figure, when the process starts, the projection control unit 12 performs a process of reading the pattern light data from the storage unit 13 (S111).

[0053] In the present embodiment, as a method for analyzing the correspondence relationship between the camera 31 and the projector 35, the spatial coding method (or spatial encoding method) is adopted, and the pattern light data includes both the pattern light data for horizontal coding and the pattern light data for vertical coding. The pattern for horizontal coding is a vertical stripe pattern (vertical pattern) in which bright and dark parts are periodically repeated, and the pattern for vertical spatial coding is a horizontal stripe pattern (horizontal pattern) in which bright and dark parts are periodically repeated (see FIG. 5).

[0054] Note that the method for analyzing the correspondence between the camera 31 and the projector 35 is not limited to the spatial coding method. Therefore, for example, other methods such as the phase shift method may be adopted.

[0055] After the reading process, in a state where the pattern light is not projected, the camera 31 performs shooting, and the captured image is acquired by the image acquisition unit 15 and stored in the storage unit 13 (S112). This shooting is the shooting of a normal color image by the camera 31.

[0056] After the shooting process, the projection control unit 12 issues a projection command for the pattern light corresponding to one of the read pattern light data to the projector 35 (S113). Based on this projection command for the pattern light, the projector 35 projects the corresponding pattern light. Note that the object at this time is a checkerboard (a plate having the shades of a checkered pattern) with a known grid pitch length.

[0057] When a projection command for the pattern light to the projector 35 is issued, the projection control unit 12 performs a process of transmitting a synchronization signal to the camera 31 (S115). When receiving this synchronization signal, the camera 31 performs a process of shooting the object on which the pattern light is projected (S116).

[0058] When the shooting process is performed, the image acquisition unit 15 acquires the pattern light image captured by the camera 31 and performs a process of storing it in the storage unit 13 (S117).

[0059] After this process, it is determined whether shooting has been completed for all the read pattern lights (S118). If there is unprojected pattern light (S118NO), after performing a process of changing the pattern light to be projected (S119), a series of processes from projection to shooting are performed again (S113 to S118).

[0060] On the other hand, when shooting has been completed for all the pattern lights (S118YES), the image processing unit 16 performs a process of generating and storing a virtual camera image (S121).

[0061] Here, the virtual camera image will be described. FIG. 4 is an explanatory diagram of the virtual camera image. In the present embodiment, the virtual camera image is a logical image that would be observed by a camera if the camera were disposed at the position of the projector 35. The virtual camera image can also be rephrased as a projector image for convenience.

[0062] As is apparent from the figure, a point P on the object corresponds to a predetermined point on a predetermined two-dimensional coordinate system (x c , y c ) related to the camera image, that is, the intersection of the line segment connecting the camera position (B) and the point P and the imaging plane. On the other hand, it corresponds to a predetermined point on a predetermined two-dimensional coordinate system (x p , y p ) related to the virtual camera image, that is, the intersection of the line segment connecting the projector position (A) and the point P and the projection plane. At this time, a certain parallax occurs between the points on each coordinate system.

[0063] FIG. 5 is an explanatory diagram regarding the generation of the virtual camera image. As already described, the storage unit 13 stores a plurality of pattern images obtained by projecting pattern light of a horizontal pattern and a vertical pattern, and a color image when the pattern light is not projected (the upper part on the right side of the figure).

[0064] In this state, the image processing unit 16 performs decoding processing (decryption processing) on the image including the horizontal pattern light and the image including the vertical pattern light. As various methods known to those skilled in the art can be adopted as the method of the decoding processing. As a result of this decoding processing, decoding information regarding the vertical direction position is obtained from the image including the horizontal pattern light, and decoding information regarding the horizontal direction position is obtained from the image including the vertical pattern light (the lower part on the right side of the figure).

[0065] Thereafter, the image processing unit 16 generates a corresponding virtual camera image that would be observed at the projector position based on the color image captured when the pattern light is not projected, the vertical decoding information, and the horizontal decoding information.

[0066] After the generation and storage processing of the virtual camera image are completed, a determination process for a predetermined end condition is performed (S122). The end condition is, for example, the number of shootings.

[0067] When the predetermined end condition is satisfied (S122 YES), the generation process of the pair image ends.

[0068] On the other hand, when the predetermined end condition is not satisfied (S122 NO), a process of determining whether a signal indicating the completion of the shooting preparation has been received is performed (S123). The image processing unit 16 performs standby processing until the shooting preparation is completed (S123 NO). Here, the shooting preparation means changing the position and orientation of the checkerboard for shooting. In this way, by changing the position and orientation of the checkerboard, a camera captured image and a virtual camera image corresponding thereto are generated for each position and orientation of the checkerboard. Further, the completion signal of the shooting preparation is a signal indicating that the shooting preparation is completed, and may be, for example, a signal input by the photographer via the input reception unit 11.

[0069] When it is detected that the shooting preparation is completed (S123 YES), the series of shooting processes (S112 to S122) are executed again.

[0070] FIG. 6 is an example of an image generated and stored by the pair image generation process (S11). As is clear from the figure, for each position and orientation of the checkerboard by the pair image generation process, a camera captured image and a corresponding virtual camera image are generated and stored in pairs.

[0071] Returning to FIG. 2, when the pair image generation process is completed, based on the camera captured image and the virtual camera image, distortion parameters k1, k2, k3, p1, p2 for correcting distortion caused by the optical system of the camera 31 or the projector 35, for example, barrel distortion, are calculated respectively (see also Equation 1 described later), and a process of storing them is performed (S13).

[0072] As is apparent to those skilled in the art, this distortion correction parameter can be obtained by a known method by detecting the positions of the feature points of the chessboard, for example, the positions of the corners of the squares forming the grid (see, for example, "A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, Zhang, 22(11):1330-1334, 2000.).

[0073] After the completion of the distortion correction process, the image processing unit 16 performs a process of identifying the three-dimensional positional and postural relationship between the camera 31 and the projector 35 from the pairs of feature points of the chessboard (S14). More specifically, in the present embodiment, an optimization calculation known to those skilled in the art is performed from the pairs of feature points of the chessboard, and a process of identifying the translation and rotation matrices of the projector 35 when the position and posture of the camera 31 are taken as the origin is performed.

[0074] When the process of identifying the position and posture is completed, the image processing unit 16, from the identified three-dimensional positional relationship between the camera 31 and the projector 35, obtains the projection transformation parameters (H in Equation 2) for the camera 31 used for the parallelization transformation 11 ~H 33 ), and the projection transformation parameters (H in Equation 2) for the projector 35 11 ~H 33 ), calculates them, and stores them respectively (S15).

[0075] Thereafter, the image processing unit 16 performs a process of generating and storing a first two-dimensional look-up table (2DLUT) (S16).

[0076] FIG. 7 is a detailed flowchart of the process of generating the first two-dimensional look-up table. As is clear from the figure, when the process starts, the image processing unit 16 initializes variables i and j for specifying pixels (S161). Thereafter, for the coordinates (i, j) of the pixel of interest on the camera-captured image, a distortion correction process is performed, and the coordinates (x c ', y cPerform a process of calculating ')'.

[0077] The distortion is the distortion caused by the optical system of the camera 31 or the like, for example, barrel distortion. This type of distortion can be corrected by a known method known to those skilled in the art. In the present embodiment, as an example, by solving the following mathematical formula 1, the coordinates (x, y) (left figure) of the pixel of interest before correction are calculated from the corresponding coordinates (x', y') (right figure) after distortion correction. As described above, the parameters of k1, k2, k3, p1, and p2 have already been acquired by photographing the chessboard.

[0078]

Equation

[0079] FIG. 8 is a conceptual diagram regarding the correction of barrel distortion. As is clear from the figure, the grid on the left side of the figure is distorted so as to bulge near the center. In this state, when the barrel distortion correction process is performed, the coordinates (x, y) of the pixel of interest before correction on the left side of the figure correspond to the coordinates (x c ', y c ) on the right side of the figure after distortion correction. In this way, the distortion correction process is executed.

[0080] After the distortion correction process, the image processing unit 16 performs a parallelization conversion process on the corresponding coordinates (x c ', y c ) after distortion correction, and performs a process of calculating the coordinates (x c '', y c '') after parallelization conversion (S163).

[0081] This parallelization conversion process can be performed by a known method known to those skilled in the art. In the present embodiment, as an example, a projective transformation is performed using the projective transformation matrix (H 11 ~H 33 ) for the camera already obtained so that the epipolar line is horizontal.

[0082]

Equation

[0083] FIG. 9 is a conceptual diagram related to the parallelization conversion process. As is clear from the figure, by the projective transformation, the four coordinates (x c ', y c ') on the left side of the figure before correction are associated with the four coordinates (x c '', y c '') on the right side of the figure.

[0084] After the parallelization conversion is completed, the image processing unit 16 performs a process of storing the corresponding coordinates (x c '', y c '') after the correction transformation at the position of the i-th row and j-th column of the two-dimensional look-up table in the storage unit 13 (S165).

[0085] Thereafter, the image processing unit 16 performs a process of determining whether the variable i is equal to the maximum value i_max of i (S166). If the variable i is not yet the maximum value (S166NO), a process of incrementing the variable i by 1 is performed (S167), and the series of processes are performed again (S162 to S166).

[0086] On the other hand, when the variable i is equal to the maximum value i_max (S166YES), the image processing unit 16 initializes the variable i (S169) and performs a determination process as to whether the variable j is equal to the maximum value j_max (S171).

[0087] If the variable j is not yet the maximum value (S171NO), a process of incrementing the variable j by 1 is performed (S168), and the series of processes are performed again (S162 to S171). On the other hand, when the variable j is equal to the maximum value j_max (S171YES), the process ends.

[0088] FIG. 10 is an explanatory diagram of the first two-dimensional look-up table according to the present embodiment generated by a series of processes. As is clear from the figure, in this table, one series is the x coordinate (horizontal coordinate) of the pixel before correction, and the other series is the y coordinate of the pixel before correction, and the x and y coordinates (x c'', y c stores '').

[0089] That is, by using this two-dimensional lookup table, for each pixel of the image acquired by the camera 31, the coordinate information (x c '', y c '') can be immediately referred to.

[0090] According to such a two-dimensional lookup table, for each pixel of the image before correction, the coordinates after correction can be obtained quickly.

[0091] Returning to FIG. 2, after the generation process of the first two-dimensional lookup table (S16), next, the generation process and the storage process of the second two-dimensional lookup table are performed (S18).

[0092] FIG. 11 is a detailed flowchart of the generation process of the second two-dimensional lookup table. As is clear from the figure, when the process starts, the image processing unit 16 performs initialization processing of variables i and j for designating pixels (S181). After that, for the target pixel coordinates (i, j) on the virtual camera image, a distortion correction process is performed, and the corresponding coordinates (x p ', y p ') after distortion correction are calculated (S182).

[0093] Note that the distortion is the distortion caused by the optical system of the projector 35 or the like, for example, barrel distortion. This type of distortion can be corrected by a known method known to those skilled in the art, and in this embodiment, it is corrected by the same method (S162) as the process for the camera image.

[0094] After the distortion correction process, the image processing unit 16 performs a parallelization conversion process on the corresponding coordinates (x p ', y p ') after distortion correction, and calculates the coordinates (x p '', y p ') after parallelization conversion (S183).

[0095] This parallelization conversion process can be performed by a well-known method known to those skilled in the art. In this embodiment, correction is performed by the same method (S163) as that for the camera image.

[0096] After the parallelization conversion is completed, the image processing unit 16 performs a process of storing the corrected corresponding coordinates (x p '', y p '') at the position of the i-th row and j-th column of the two-dimensional look-up table in the storage unit 13 (S185).

[0097] Thereafter, the image processing unit 16 performs a process of determining whether the variable i is equal to the maximum value i_max of i (S186). If the variable i is not yet the maximum value (S186NO), a process of incrementing the variable i by 1 is performed (S187), and the series of processes are performed again (S182 to S186).

[0098] On the other hand, when the variable i is equal to the maximum value i_max (S186YES), the image processing unit 16 initializes the variable i (S189) and performs a determination process as to whether the variable j is equal to the maximum value j_max (S191).

[0099] If the variable j is not yet the maximum value (S191NO), a process of incrementing the variable j by 1 is performed (S188), and the series of processes are performed again (S182 to S191). On the other hand, when the variable j is equal to the maximum value j_max (S191YES), the format conversion process of the two-dimensional look-up table is performed (S192).

[0100] FIG. 12 is a conceptual diagram of the format conversion process. As is clear from the figure, the two-dimensional look-up table before the format conversion has one series as x p , and the other series as y p , and stores the corrected coordinates (x p '', y p '') at the positions specified by those series (FIG. (A) therein).

[0101] In the format conversion process, first, based on the two-dimensional lookup table, for each pixel (x p , y p ), a pair list of the pixel and the corrected coordinates (x p '', y p '') corresponding to the pixel is generated (Figure (B) in the same figure). At this time, since y p is not used in the subsequent processing, it may be deleted.

[0102] Based on this pair list, a two-dimensional graph is generated with the origin as O, one series as x p , and the other series as y p '' (Figure (C) in the same figure). At this time, the value of x p '' is associated with each point on the graph.

[0103] After that, the image processing unit 16 generates triangles connecting three adjacent points, for example, by the Delaunary triangulation method (Delaunay triangulation method) known to those skilled in the art (Figure (D) in the same figure).

[0104] After generating this triangle, the image processing unit 16 fits an integer grid to both series of x p , y p '', and calculates the value of x p '' at each integer coordinate by linear interpolation of the three points forming the triangle (Figure (E) in the same figure). Using this linearly interpolated value, a two-dimensional lookup table with one series as x p , and the other series as y p '', that is, a second two-dimensional lookup table, is generated (Figure (F) in the same figure).

[0105] Figure 13 is a conceptual diagram of the two-dimensional lookup table before and after format conversion. As is clear from the same figure, before format conversion, as shown on the left side of the same figure, one series of the two-dimensional lookup table is x p , and the other series is y p . At each coordinate, the corrected coordinates (x p '', y p'') is stored. On the other hand, after the format conversion, as shown on the right side of the figure, one series of the two-dimensional look-up table is x p , and the other series is y p '', and for each coordinate, the corrected x-coordinate x p '' is stored.

[0106] That is, by using the second two-dimensional look-up table, based on the x-coordinate of the pixel of interest in the virtual camera image and the corresponding y-coordinate (y p '') after the distortion correction and parallelization conversion processing, the corresponding x-coordinate (x p '') after the distortion correction processing and parallelization conversion processing can be immediately referred to. That is, a high-speed reference to the corresponding x-coordinate (x p '') after the distortion correction processing and parallelization conversion processing becomes possible.

[0107] Through the above processing, the first look-up table and the second look-up table are generated and stored in the preprocessing.

[0108] (1.2.2 3D measurement processing) Next, the 3D measurement processing performed after the preprocessing will be described.

[0109] FIG. 14 is a general flowchart of the 3D measurement processing. As is clear from the figure, when the processing starts, the input reception unit 11 enters the standby state for receiving the measurement start signal (S31 NO). In this state, when the measurement start signal is received (S31 YES), the subsequent processing is executed. Note that the measurement start signal may be transmitted from a higher-level system of the 3D measurement system 100 or may be generated by an input from the user.

[0110] After receiving the measurement start signal, imaging processing is performed by the camera 31, and the captured image is acquired by the image acquisition unit 15 (S32). This imaging is the normal color image imaging by the camera 31.

[0111] After that, the projection control unit 12 performs generation processing of the pattern light image (S34).

[0112] FIG. 15 is a detailed flowchart of the generation processing of the pattern light image. As is clear from the figure, when the processing starts, the projection control unit 12 reads out pattern light data from the storage unit 13 (S341). At this time, the read pattern light data is only the data related to the vertical pattern that performs spatial encoding in the x-axis direction.

[0113] When the readout processing of the pattern light data is completed, the projection control unit 12 issues a projection command for the pattern light corresponding to one of the read pattern light data to the projector 35 (S342). Based on this projection command for the pattern light, the projector 35 projects the corresponding pattern light.

[0114] When the pattern light projection command to the projector 35 is issued, the projection control unit 12 performs processing to transmit a synchronization signal to the camera 31 (S344). When receiving this synchronization signal, the camera 31 performs processing to photograph the pattern light projected onto the object (S345).

[0115] When the photographing processing is performed, the image acquisition unit 15 acquires the pattern light image photographed by the camera 31 and performs processing to store it in the storage unit 13 (S346).

[0116] After this processing, it is determined whether or not photographing has been completed for all the read pattern lights (S348). If there is unprojected pattern light (S348 NO), after performing processing to change the pattern light to be projected (S349), a series of processing from projection to photographing is performed again (S342 to S348). On the other hand, if photographing has been completed for all the pattern lights (S348 YES), the processing ends.

[0117] Returning to FIG. 14, when the generation processing of the pattern light image is completed, predetermined image processing is performed based on the pattern light image (S36).

[0118] FIG. 16 is a detailed flowchart related to image processing. As is clear from the figure, when the processing starts, the image processing unit 16 performs an initialization process of an integer variable i, for example, a process of setting i = 0 (S361).

[0119] After the initialization process, the image processing unit 16 performs a process of reading out the first lookup table and the second lookup table stored in the storage unit 13 (S362).

[0120] Thereafter, the image processing unit 16 performs a decoding process on the pattern light image composed of vertical patterns (S363).

[0121] FIG. 17 is a conceptual diagram of the decoding process. As is clear from the figure, by decoding the pattern light image composed of vertical patterns shown on the left side of the figure, decoding information that enables identification of the horizontal (x-axis) direction position on the virtual camera image corresponding to the pixel coordinates of interest in the camera-captured image can be obtained (as an example, the gradation on the right side of the figure). Note that in the present embodiment, since only the pattern light related to the vertical pattern is projected, it should be noted that vertical direction information cannot be obtained even when the decoding process is performed.

[0122] Thereafter, using the first lookup table with the i-th pixel as the pixel of interest, the coordinates (x c , y c ) corresponding to the pixel of interest, and after performing distortion correction and parallelization conversion processing, the corrected coordinates (x c '', y c '') are obtained.

[0123] After obtaining the corrected coordinates regarding the camera-captured image, the image processing unit 16, based on the decoded information, identifies the horizontal (x-axis) direction position (x c , y c ) on the virtual camera image corresponding to the coordinates (x p ) of the pixel of interest (S366).

[0124] Here, the coordinate position y in the corrected vertical (y-axis) direction in the camera-captured imagec '' is the corrected vertical (y-axis) coordinate y in the virtual camera image through the parallelization conversion process p '' becomes equal to (y c '' = y p ''). Utilizing this, after the above-mentioned specific process (S366), the image processing unit 16 uses the second look-up table to determine the horizontal (x-axis) position (x p ) on the virtual camera image and the corrected vertical (y-axis) coordinate position y c '' (= y p '') to obtain the corrected horizontal (x-axis) value (x p '') related to the corresponding virtual camera image (S367).

[0125] That is, as one series of the second look-up table (refer to the right figure in FIG. 13), the horizontal (x-axis) position (x p ) of the pixel of interest on the virtual camera image is specified, and as the other series, the corrected vertical (y-axis) position y c '' (= y p '') of the pixel of interest is specified, and the corrected horizontal (x-axis) value (x p '') related to the corresponding virtual camera image stored at the position specified thereby is obtained.

[0126] After the above processing, the image processing unit 16 calculates the disparity information d from each corrected horizontal (x-axis) coordinate (x c '' and x p ''). The disparity information is calculated simply as the difference between x c '' and x p '' in this embodiment (= x c '' - x p '').

[0127] After calculating the disparity information d, the image processing unit 16 performs a process of determining whether the process has been completed for all pixels (S369). If the process has not been completed for all pixels yet (S369 NO), the variable i is incremented, and a series of processes (S365 to S368) are performed for another pixel. On the other hand, if the process has been completed for all pixels (S369 YES), the process ends. As a result, the disparity information d is calculated for each pixel.

[0128] FIG. 18 is a conceptual diagram regarding the flow of generating the disparity information d. In the figure, the right column shows the process for the acquired image from the camera 31, and the left column shows the process for the logical image that would be acquired by the virtual camera at the position of the projector 35.

[0129] As is clear from the right column of the figure, when the process starts, vertical pattern light is projected onto the object 5, and a pattern light image is generated (S34). For this pattern light image, the image processing unit 16 uses the first look-up table to obtain the corrected corresponding pixel coordinates (x c '', y c '') after performing distortion correction processing and parallelization conversion processing (S362, S365) (the lowermost part of the right column in the figure).

[0130] On the other hand, as is clear from the left column of the figure, the image processing unit 16 decodes the pattern light image to obtain the value (x p ) in the horizontal (x-axis) direction corresponding to the pixel of interest and related to the virtual camera image (S362, S363, S366).

[0131] After that, using the fact that the corrected vertical (y-axis) coordinate y c '' in the camera-captured image is equal to the corrected vertical (y-axis) coordinate y p '' in the virtual camera image, the horizontal (x-axis) direction position (x p ) on the virtual camera image and the corrected vertical (y-axis) direction coordinate position y cBased on '', the corrected value in the horizontal (x-axis) direction (x p '') related to the corresponding virtual camera image is obtained (S367) (the bottom row in the left column of the figure).

[0132] Finally, as shown in the bottom row in the center of the figure, the image processing unit 16 calculates the disparity information d from the coordinates (x c '' and x p '') in the corrected horizontal (x-axis) direction (S368).

[0133] According to such a configuration, by simply projecting the pattern light subjected to spatial encoding in one direction, corrected projector coordinates considering the distortion generated in the projector can be obtained. Thereby, while correcting the distortion generated in the optical system of the projector by image processing, high-speed three-dimensional measurement can be realized.

[0134] Returning to FIG. 14, when the image processing is completed, the image processing unit 16 performs distance information generation processing (S37). In the present embodiment, in addition to the disparity information d, using the known parameters (focal length f and the distance B between the camera 31 and the projector 35) stored in the storage unit 13, the distance from the camera 31 or the projector 35 to the object 5 is calculated.

[0135] FIG. 19 is a principle diagram regarding the generation of distance information. As shown in the figure, a point on the object 5 is point P, the projector 35 or the virtual camera position is point Q, the camera 31 position is point R, and the distance from point Q or point R to the image sensor 311 is f. Also, the distance from the intersection of the line extended parallel to the line segment PQ from point R and the image sensor 311 plane to the intersection of the line segment PR and the image sensor 311 plane is the already obtained disparity d. Furthermore, the distance of the baseline from point Q to point R is B.

[0136] In this state, the distance Z from point Q or point R to point P is calculated as the value obtained by dividing the product of the distance B and the focal length f by the disparity d, as shown in Equation 3 below.

[0137]

Equation

[0138] That is, by obtaining the parallax information d, the distance information Z to the object can be calculated. Further, since such distance information Z can be generated for each pixel, a so-called depth map can be generated.

[0139] Returning to FIG. 14, when the generation process of the distance information d is completed, next, the generation and output process of the distance image are performed (S38).

[0140] In the present embodiment, the image processing unit 16 generates point cloud data in a three-dimensional space using the normal image and the depth map captured in the shooting process (S32). Thereafter, the output unit 17 renders the point cloud data together with the depth map and outputs it to a display (not shown) or the like.

[0141] According to such a configuration, it is possible to provide distance information within the camera imaging region and to intuitively present the distance information to the user.

[0142] FIG. 20 is a conceptual diagram related to the generation of three-dimensional point cloud data. As is clear from the figure, by performing image processing so as to apply depth map information (lower left in the figure) to a normal captured image (upper left in the figure) by the camera 31, a three-dimensional image as shown on the right side of the figure can be provided.

[0143] In the present embodiment, image information is output. However, the present invention is not limited to such a configuration, and simply the parallax information d or the distance information Z may be output.

[0144] (2. Second Embodiment) The second embodiment of the present invention will be described while referring to the accompanying drawings. In the following, the same reference numerals are given to the same configurations as those in the first embodiment.

[0145] (2.1 System Configuration) FIG. 21 is an overall configuration diagram of the three-dimensional measurement system 100' according to the present embodiment. As is clear from the figure, the three-dimensional measurement system 100' according to the present embodiment also includes a controller unit 1' and a sensor head unit 3' connected to the controller unit 1'. Therefore, in terms of generating three-dimensional information of the object 5 by using these configurations, it has a configuration similar to that of the first embodiment.

[0146] However, the sensor head unit 3' according to the present embodiment is different from the configuration according to the first embodiment in that it further includes a temperature sensor 37 for measuring the temperature of an optical system such as a lens of the projector 35. More specifically, the temperature sensor 37 is attached to a resin plate 38 having the same thermal conductivity as the lens, which is attached to the aluminum lens barrel of the projector 35. That is, it is indirectly attached to the lens barrel via the resin plate 38.

[0147] By attaching the temperature sensor 37 to the resin plate 38 having the same thermal conductivity as the lens in this way, the temperature sensor 37 can measure a temperature similar to that of the lens. As a result, the thermal expansion of the optical system such as the lens can be grasped more accurately, and three-dimensional measurement can be performed with high accuracy. Note that the temperature sensor 37 may be directly attached to the lens barrel.

[0148] In addition, the sensor information acquired by the temperature sensor 37 is stored in the storage unit 131 and is used in the processing in the image processing unit 161 as described later.

[0149] (2.2 System operation) (2.2.1 Preprocessing) The preprocessing in the second embodiment is substantially the same as the preprocessing in the first embodiment. However, it is different in that a second look-up table is created while changing the temperature conditions acquired by the temperature sensor 37 and stored in the storage unit 131 together with the temperature conditions.

[0150] FIG. 22 is a conceptual diagram of a second look-up table group (3D look-up table (3DLUT)) generated in the present embodiment. As is clear from the figure, in the example of the figure, the second look-up table is generated every 10 degrees. Similar to the first embodiment, each look-up table has one series as the x coordinate (horizontal coordinate x p ) of the pixel before correction, and the other series as the y coordinate (vertical coordinate y p '') of the pixel after correction, and stores the x coordinate (x p '') after correction at the position specified by those coordinates.

[0151] That is, by selectively using a two-dimensional look-up table according to the temperature, based on the x coordinate (x p ) of the pixel before correction and the y coordinate (y p '') of the pixel after correction, for each pixel of the logical image of the virtual camera, the x coordinate (x p '') which is the result of performing distortion correction processing and parallelization conversion processing can be immediately obtained. According to such a two-dimensional look-up table, for each pixel of the image before correction, the coordinates after correction can be obtained at high speed.

[0152] (2.2.2 3D measurement processing) The 3D measurement processing in the second embodiment is also substantially the same as the 3D measurement processing in the first embodiment.

[0153] FIG. 23 is a general flowchart of the 3D measurement processing according to the second embodiment. As is clear from the figure, after receiving a measurement start signal (S31), the flow of performing camera shooting (S32) and generating a pattern light image (S34) is the same as that according to the first embodiment. However, the content of the image processing according to the second embodiment is different from the content according to the first embodiment.

[0154] FIG. 24 is a detailed flowchart related to the image processing according to the present embodiment. As is apparent from the figure, the processing from the initialization process (S361) of the variable i to the generation (S368, S369) of the disparity information d for each pixel is substantially the same as that according to the first embodiment. However, it is different in that the processing using the temperature sensor 37 is further added.

[0155] After the initialization process (S361) of the variable i, the image processing unit 161 acquires the temperature of the optical system of the projector 35 through the temperature sensor 37 (S561). Thereafter, the image processing unit 161 performs a process of reading out the second look-up table generated under the temperature condition closest to the acquired temperature from the storage unit 131 (S562).

[0156] Based on this read second look-up table, the value (X p '') in the horizontal direction (x-axis direction) of the corrected coordinates in the virtual camera is acquired (S367).

[0157] In the present embodiment, a process of reading out the second look-up table generated under the temperature condition closest to the acquired temperature is performed, but other methods may be adopted. For example, two second look-up tables close to the acquired temperature may be read out, and a look-up table with values interpolated by linear interpolation may be generated from them.

[0158] According to such a configuration, it is possible to acquire the projector coordinates or the virtual camera coordinates after performing appropriate correction processing according to the temperature of the optical system related to the projector.

[0159] Also, it is possible to acquire appropriate corrected virtual camera coordinates, that is, corrected projector coordinates, at any temperature by linear interpolation.

[0160] (3. Modification Example)

[0161] The present invention can be implemented with various modifications.

[0162] For example, when it is guaranteed that parallelism is achieved between the image sensor surface of the camera 31 and the imaging surface of the virtual camera at the position of the projector 35, only the distortion correction process may be performed without performing the parallelization conversion process.

[0163] FIG. 25 shows a system configuration when the camera 31 and the projector 35 are mechanically or optically parallelized. In such a configuration, there is no need to perform the parallelization conversion process.

[0164] In the above-described embodiment, pattern light subjected to spatial encoding in the horizontal direction (x-axis direction) is projected, and parallax information is calculated from the difference in position coordinates in the horizontal direction (x-axis direction). However, the direction in which the spatial encoding of this pattern light is performed and the direction in which the difference is taken for parallax are relatively determined by the positional relationship between the camera 31 and the projector 35.

[0165] Therefore, for example, when the camera 31 and the projector 35 are arranged vertically, the direction in which the spatial encoding of the pattern light is performed and the direction in which the difference is taken for parallax may be determined based on the direction parallel to the virtual line connecting them.

[0166] In the above-described embodiment, the coordinates (x c '', y c '') of each pixel of the corrected camera image and the corrected coordinates (x p '') on the virtual camera image are calculated. However, the present invention is not limited to such a configuration. Therefore, for example, it may not be in the form of a two-dimensional table, but may simply be in a form in which corresponding values are read out. Also, calculations may be performed sequentially.

[0167] The embodiments of the present invention have been described above. However, the above embodiments merely show a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. Also, the above embodiments can be appropriately combined as long as there is no contradiction.

Industrial Applicability

[0168] The present invention can be used in industries manufacturing at least three-dimensional measurement systems and the like.

Explanation of Signs

[0169] 1 Controller unit 11 Input reception unit 12 Projection control unit 13 Storage unit 15 Image acquisition unit 16 Image processing unit 17 Output unit 3 Sensor head unit 31 Camera 311 Image sensor 312 Objective lens 35 Projector 351 Light source 352 Plane wave conversion lens 353 Mirror 355 DMD 356 Relay lens 357 Objective lens 37 Temperature sensor 5 Object 100 Three-dimensional measurement system

Claims

1. A projector that projects striped pattern light in which light and dark parts are periodically repeated with respect to an object in a first direction, A camera that captures the pattern light projected onto the object to generate a captured image, A corrected camera coordinate acquisition unit that performs distortion correction processing and parallelization conversion processing related to the camera on a target pixel on the captured image, thereby obtaining corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction, A decoding processing unit that performs decoding processing related to the pattern light on the captured image and obtains decoding information related to the first direction, A corresponding projector coordinate acquisition unit that obtains corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the position of the projector based on the decoding information, A corrected projector coordinate acquisition unit that obtains corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction and parallelization conversion processing related to the projector on the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction, A distance information generation unit that generates distance information from the camera or the projector to the object based on parallax information obtained from the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction, and The three-dimensional measurement system, wherein the first direction is a direction substantially parallel to a virtual line connecting the projector and the camera.

2. The three-dimensional measurement system according to claim 1, wherein in the corrected projector coordinate acquisition unit, the corrected projector coordinates related to the first direction are stored in a two-dimensional lookup table in which the corrected camera coordinates related to the second direction are in a first series and the corresponding projector coordinates related to the first direction are in a second series.

3. The two-dimensional lookup table is generated based on a result of projecting striped pattern light in which light and dark parts are periodically repeated with respect to a first direction and striped pattern light in which light and dark parts are periodically repeated with respect to a second direction onto a predetermined object and capturing the projected pattern light. The three-dimensional measurement system according to claim 2.

4. A temperature sensor that directly or indirectly measures the temperature of the optical system related to the projector, A storage unit that stores a plurality of two-dimensional look-up tables for each temperature, which generate corrected camera coordinates related to the second direction in a first series and corresponding projector coordinates related to the first direction in a second series, A selective reading unit that reads out the corresponding two-dimensional look-up table for each temperature according to the temperature measured by the temperature sensor, and further includes, The corrected projector coordinate acquisition unit acquires the corrected projector coordinates related to the first direction by using the read two-dimensional look-up table for each temperature instead of the two-dimensional look-up table. The three-dimensional measurement system according to claim 2.

5. The corrected projector coordinates related to the first direction are generated by performing interpolation processing using values stored in the two-dimensional look-up table for each temperature corresponding to temperatures before and after the temperature measured by the temperature sensor. The three-dimensional measurement system according to claim 4.

6. The interpolation processing is linear interpolation processing. The three-dimensional measurement system according to claim 5.

7. The pattern light is pattern light related to a spatial coding method. The three-dimensional measurement system according to claim 1.

8. The pattern light is pattern light related to a phase shift method. The three-dimensional measurement system according to claim 1.

9. A projector that projects a striped pattern light in which bright and dark parts are periodically repeated in a first direction onto an object, A camera that captures the pattern light projected onto the object to generate a captured image, For a target pixel on the captured image, by performing distortion correction processing and parallelization conversion processing related to the camera, corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction are obtained. A corrected camera coordinate acquisition unit, A decoding processing unit that performs decoding processing related to the pattern light on the captured image and acquires decoding information related to the first direction, Based on the decoding information, on a virtual logical image observed in a virtual camera arranged at the position of the projector, corresponding to the target pixel and related to the first direction, a corresponding projector coordinate acquisition unit that acquires corresponding projector coordinates. Based on the corrected camera coordinates regarding the second direction and the corresponding projector coordinates regarding the first direction, a corrected projector coordinate acquisition unit that acquires corrected projector coordinates regarding the first direction corresponding to the result of performing distortion correction and parallelization conversion processing on the corresponding projector coordinates with respect to the projector; A distance information generation unit that generates distance information from the camera or the projector to the object based on the corrected camera coordinates regarding the first direction and the parallax information obtained from the corrected projector coordinates regarding the first direction; A three-dimensional measurement device, wherein the first direction is a direction substantially parallel to an imaginary line connecting the projector and the camera.

10. A control method for a three-dimensional measurement system including a projector that projects striped pattern light in which bright and dark portions are periodically repeated in a first direction onto an object, and a camera that captures the pattern light projected onto the object to generate a captured image, the method comprising: A corrected camera coordinate acquisition step of performing distortion correction processing and parallelization conversion processing regarding the camera on a target pixel on the captured image to obtain corrected camera coordinates corresponding to the target pixel and regarding the first direction and a second direction orthogonal to the first direction; A decoding processing step of performing decoding processing regarding the pattern light on the captured image to obtain decoded information regarding the first direction; A corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates corresponding to the target pixel and regarding the first direction on a virtual logical image observed in a virtual camera disposed at the position of the projector based on the decoded information; A corrected projector coordinate acquisition step of obtaining corrected projector coordinates regarding the first direction corresponding to the result of performing distortion correction and parallelization conversion processing on the corresponding projector coordinates with respect to the projector based on the corrected camera coordinates regarding the second direction and the corresponding projector coordinates regarding the first direction; A distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates regarding the first direction and the parallax information obtained from the corrected projector coordinates regarding the first direction. A control method, wherein the first direction is a direction substantially parallel to an imaginary line connecting the projector and the camera.

11. A control program for a three-dimensional measurement system including a projector that projects striped pattern light in which bright and dark portions are periodically repeated with respect to an object in a first direction, and a camera that captures the pattern light projected onto the object to generate a captured image, the control program comprising: A corrected camera coordinate acquisition step of performing distortion correction processing and parallelization conversion processing related to the camera on a target pixel in the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; A decoding process step of performing decoding processing related to the pattern light on the captured image to obtain decoding information related to the first direction; A corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates that correspond to the target pixel and are related to the first direction on a virtual logical image observed in a virtual camera disposed at the position of the projector based on the decoding information; A corrected projector coordinate acquisition step of obtaining corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction and parallelization conversion processing related to the projector on the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction; A distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and parallax information obtained from the corrected projector coordinates related to the first direction. A control program, wherein the first direction is a direction substantially parallel to an imaginary line connecting the projector and the camera.

12. A projector that projects striped pattern light in which bright and dark portions are periodically repeated with respect to an object in a first direction, and a camera that captures the pattern light projected onto the object to generate a captured image, a corrected camera coordinate acquisition unit that performs distortion correction processing and parallelization conversion processing related to the camera on a target pixel in the captured image to obtain corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction; A decoding processing unit that performs decoding processing on the captured image with respect to the pattern light and obtains decoding information regarding the first direction; A corresponding projector coordinate acquisition unit that, based on the decoding information, obtains corresponding projector coordinates regarding the first direction that are on a virtual logical image observed by a virtual camera disposed at the position of the projector and correspond to the pixel of interest; A corrected projector coordinate acquisition unit that obtains corrected projector coordinates regarding the first direction corresponding to a result of performing distortion correction and parallelization conversion processing on the projector with respect to the corresponding projector coordinates based on the corrected camera coordinates regarding the second direction and the corresponding projector coordinates regarding the first direction; A distance information generation unit that generates distance information from the camera or the projector to the object based on the corrected camera coordinates regarding the first direction and parallax information obtained from the corrected projector coordinates regarding the first direction. The three-dimensional camera, wherein the first direction is a direction substantially parallel to a virtual line connecting the projector and the camera. The first direction is a direction substantially parallel to a virtual line connecting the projector and the camera. [

13. ] A projector that projects striped pattern light in which bright and dark portions are periodically repeated with respect to an object in a first direction; A camera that captures the pattern light projected onto the object to generate a captured image; A corrected camera coordinate acquisition unit that, by performing distortion correction processing on the camera with respect to a pixel of interest on the captured image, obtains corrected camera coordinates regarding the first direction and a second direction orthogonal to the first direction that correspond to the pixel of interest; A decoding processing unit that performs decoding processing on the captured image with respect to the pattern light and obtains decoding information regarding the first direction; A corresponding projector coordinate acquisition unit that, based on the decoding information, obtains corresponding projector coordinates regarding the first direction that are on a virtual logical image observed by a virtual camera disposed at the position of the projector and correspond to the pixel of interest; A corrected projector coordinate acquisition unit that obtains corrected projector coordinates regarding the first direction corresponding to a result of performing distortion correction processing on the projector with respect to the corresponding projector coordinates based on the corrected camera coordinates regarding the second direction and the corresponding projector coordinates regarding the first direction; A distance information generation unit that generates distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the parallax information obtained from the corrected projector coordinates related to the first direction. The three-dimensional measurement system, wherein the first direction is a direction substantially parallel to an imaginary line connecting the projector and the camera. **Claim 14** A control method for a three-dimensional measurement system including a projector that projects striped pattern light in which bright and dark portions are periodically repeated in a first direction onto an object, and a camera that captures the pattern light projected onto the object to generate a captured image, the method comprising: A corrected camera coordinate acquisition step of performing a distortion correction process related to the camera on a target pixel on the captured image to acquire corrected camera coordinates corresponding to the target pixel and related to the first direction and a second direction orthogonal to the first direction. A decoding process step of performing a decoding process related to the pattern light on the captured image to acquire decoding information related to the first direction. A corresponding projector coordinate acquisition step of acquiring corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera disposed at the position of the projector based on the decoding information. A corrected projector coordinate acquisition step of acquiring corrected projector coordinates related to the first direction corresponding to a result of performing a distortion correction process related to the projector on the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction. A distance information generation step of generating distance information from the camera or the projector to the object based on the corrected camera coordinates related to the first direction and the parallax information obtained from the corrected projector coordinates related to the first direction. The control method, wherein the first direction is a direction substantially parallel to an imaginary line connecting the projector and the camera. **Claim 15** A control program for a three-dimensional measurement system including a projector that projects striped pattern light in which bright and dark portions are periodically repeated in a first direction onto an object, and a camera that captures the pattern light projected onto the object to generate a captured image, the program comprising: A correction camera coordinate acquisition step of performing distortion correction processing on the camera for a target pixel on the captured image to obtain correction camera coordinates corresponding to the target pixel and related to a first direction and a second direction orthogonal to the first direction; A decoding process step of performing decoding processing on the captured image for the pattern light to obtain decoding information related to the first direction; A corresponding projector coordinate acquisition step of obtaining corresponding projector coordinates corresponding to the target pixel and related to the first direction on a virtual logical image observed in a virtual camera arranged at the position of the projector based on the decoding information; A corrected projector coordinate acquisition step of obtaining corrected projector coordinates related to the first direction corresponding to a result of performing distortion correction processing on the projector for the corresponding projector coordinates based on the corrected camera coordinates related to the second direction and the corresponding projector coordinates related to the first direction; A distance information generation step of generating distance information from the camera or the projector to the object based on the parallax information obtained from the corrected camera coordinates related to the first direction and the corrected projector coordinates related to the first direction; and The first direction is a direction substantially parallel to a virtual line connecting the projector and the camera, the control program.

Images