3D shape measurement system, 3D shape measurement method, and 3D shape measurement program

Abstract

To provide an image processing system that can accurately obtain a three-dimensional image of an object even if the number of picked-up images is small, and can also suppress the effects of light reflection.SOLUTION: A three-dimensional shape measurement system for measuring a three-dimensional shape of an object performing known movement comprises a projection unit 1 that projects one projection image 101 having a code pattern onto the object performing known movement, an imaging unit 2 that continuously captures the object onto which the one projection image 101 is projected to generate a plurality of camera images 201, and a restoration unit 44 that sets a plurality of hypothesis points m1 to m4 on the camera line of sight for an arbitrary point p0 in the camera image and determines a hypothesis point among the plurality of hypothetical points m1 to m4 that satisfies motion constraints based on the known movement as a restoration point for the arbitrary point in question.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional shape measurement system, a three-dimensional shape measurement method, and a three-dimensional shape measurement program that measure the three-dimensional shape of an object by projecting a coded pattern image onto the object and photographing the object onto which the coded pattern image is projected. [Background technology]

[0002] Many methods have been devised to measure the 3D shape of an object by projecting a pattern image onto the object. Types of pattern images used include spot light projection and slit light projection. Spot light projection projects a laser beam onto the object, while slit light projection, also known as the light section method, is a method for determining the 3D position of an object by projecting a slit of light onto the object. Slit light projection is widely used in industrial measurement and other fields. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Brenner, C., Boehm, J. and Guehring, J.: Photogrammetric calibration and accuracy evaluation of a crosspattern stripe projector, Videometrics VI, Vol. 3641, International Society for Optics and Photonics, pp. 164-173 (1998). [Non-patent document 2] Sato, K. and Inokuchi, S.: Three-dimensional surface measurement by space encoding range imaging, Journal of Robotic Systems, Vol. 2, pp. 27-39 (1985). [Non-patent document 3] Sagawa, R., Furukawa, R. and Kawasaki, H.: Dense 3D reconstruction from high frame-rate video using a static grid pattern, IEEE transactions on pattern analysis and machine intelligence, Vol. 36, No. 9, pp. 1733-1747 (2014). [Non-patent document 4] T. Weise, B. Leibe, and LV Gool. : Fast 3D scanning with automatic motion compensation, In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, 2007. [Non-Patent Document 5] Shiba, Y., Ono, S., Furukawa, R., Hiura, S., & Kawasaki, H.: Temporal Shape Super-Resolution by Intra-frame Motion Encoding Using High-fpsStructured Light, In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017 (pp. 115-123). Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional projection methods, a large number of captured images are required to reconstruct an image of an object, and image reconstruction is often affected by light reflections from the object and surrounding objects.

[0005] Therefore, the present invention aims to provide an image processing system, an image processing device, and a program that can accurately obtain a three-dimensional image of an object even with a small number of captured images, and that can suppress the effects of light reflection. [Means for solving the problem]

[0006] A three-dimensional shape measurement system according to one embodiment of the present invention is a three-dimensional shape measurement system for measuring the three-dimensional shape of an object with known movement, and is configured to include a projection unit that projects a single projection image having a coded pattern onto the object with known movement, a photographing unit that continuously photographs the object with known movement onto which the single projection image is projected to generate multiple camera images, and a restoration unit that sets multiple hypothesis points on the camera line of sight for any point on the camera image, and determines a hypothesis point from among the multiple hypothesis points that satisfies a motion constraint based on the known motion as a restoration point for the any point.

[0007] With this configuration, compared to measurement systems that project only one-dimensional lines using a single projection image, such as the slit light projection method, projecting a single projection image containing a coded pattern as described above allows for dense measurement of the three-dimensional shape of an object even with a relatively small number of captured images required for measurement. Furthermore, the orthogonality characteristic of the random pattern enables measurement of three-dimensional shapes that are robust against interreflection and other optical phenomena. While measurement systems that project only one-dimensional lines, such as the slit light projection method, are not robust against optical phenomena such as interreflection and halation, using a projection image of a coded pattern as described above improves robustness against optical phenomena such as interreflection and halation.

[0008] In the above three-dimensional shape measuring system, the coded pattern may be a random slit pattern in which a plurality of slits are randomly arranged.

[0009] Since the random slit pattern has few boundaries between lines and is less affected by adjacent lines, this configuration enables measurement of the three-dimensional shape of an object particularly well in scenes where optical phenomena are not occurring strongly.

[0010] In the above three-dimensional shape measurement system, the extension direction of the slit may be a direction perpendicular to the known movement.

[0011] This configuration allows a sufficient pattern of brightness change to be obtained in multiple images taken while the object is moving in a known manner.

[0012] In the above three-dimensional shape measurement system, the random pattern may be a random dot pattern in which a plurality of dots are randomly arranged in a two-dimensional pattern.

[0013] This configuration has the advantages of being particularly robust against interreflection and making it difficult for noise points to be restored.

[0014] In the above-described three-dimensional shape measurement system, the restoration unit may move the plurality of hypothesis points using a motion parameter that represents the known motion, and may determine the hypothesis point that maximizes the correlation between a set of luminance change patterns obtained by perspectively projecting the moved plurality of hypothesis points onto the camera image and the projection image, as satisfying the motion constraint, and may determine the hypothesis point as the restoration point.

[0015] With this configuration, when searching for a depth point on a moving object from among multiple hypothesis points, the object must move with known motion parameters as a constraint, and shape reconstruction can be performed by utilizing the maximum correlation between the pair of brightness change patterns of the projection coordinates and the shooting coordinates obtained at the hypothesis point with the correct depth value.

[0016] The above three-dimensional shape measurement system may further include a geometric constraint processing unit that excludes the initial hypothesis point set by the restoration unit and the moved hypothesis point from candidates for the restoration point if the initial hypothesis point and the moved hypothesis point do not satisfy the epipolar constraint and / or the depth constraint.

[0017] With this configuration, if multiple hypothesis points are set on the camera line of sight, the number of hypotheses will explode, resulting in extremely poor calculation efficiency and a reduced probability of obtaining a correct restoration point. However, this configuration makes it possible to search for a restoration point by limiting the number of hypothesis points using epipolar constraints and / or depth constraints.

[0018] The above-mentioned three-dimensional measurement system may further include a motion parameter acquisition unit that uses a calibration pattern instead of the object to determine the movement flow of restoration points of points on the calibration pattern obtained by the projection unit, the photographing unit, and the restoration unit, thereby determining the motion parameters.

[0019] This configuration allows us to estimate motion parameters using the same known motion as when measuring the 3D shape of an actually moving object.

[0020] A three-dimensional shape measurement method according to one embodiment of the present invention is a three-dimensional shape measurement method for measuring the three-dimensional shape of an object with known movement, and includes a projection step of projecting a projection image having a coded pattern onto the object with known movement; a photographing step of continuously photographing the object with known movement onto which the one projection image is projected to generate a plurality of camera images; and a restoration step of setting a plurality of hypothesis points on the camera line of sight for any point on the camera image, and determining a hypothesis point among the plurality of hypothesis points that satisfies a motion constraint based on the known motion as a restoration point for the any point.

[0021] This configuration also makes it possible to measure the three-dimensional shape of an object densely even with a relatively small number of captured images, and to measure the three-dimensional shape in a manner that is robust against optical phenomena such as interreflection and halation.

[0022] A three-dimensional shape measurement program according to one embodiment of the present invention is a three-dimensional shape measurement program for measuring the three-dimensional shape of an object with known movement, and is configured to be executed by an information processing device connected to a projector and a camera, causing the projector to project a single projection image having an encoded pattern onto the object with known movement, causing the camera to continuously photograph the object with known movement onto which the single projection image is projected to generate multiple camera images, setting multiple hypothetical points on the camera line of sight for any point on the camera image, and determining a hypothetical point among the multiple hypothetical points that satisfies a motion constraint based on the known motion as a restoration point for the any point.

[0023] This configuration also makes it possible to measure the three-dimensional shape of an object densely even with a relatively small number of captured images, and to measure the three-dimensional shape in a manner that is robust against optical phenomena such as interreflection and halation. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a diagram showing an application scene of a three-dimensional shape measurement system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating the principle of three-dimensional shape measurement by the three-dimensional shape measurement system according to the embodiment of the present invention. [Figure 3] FIG. 3 is a diagram for explaining the processing steps of the decoding and hypothesis point determination algorithm using motion constraints according to the embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram showing a functional configuration of the three-dimensional shape measuring system according to the embodiment of the present invention. [Figure 5] FIG. 5 is a flowchart of a three-dimensional shape measuring method according to an embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart of motion parameter estimation according to an embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating an apparatus for estimating motion parameters according to an embodiment of the present invention. [Figure 8A] FIG. 8A is a diagram showing an example of a projected image according to the embodiment of the present invention. [Figure 8B] FIG. 8B is a diagram showing an example of a projected image according to the embodiment of the present invention. [Figure 8C] FIG. 8C is a diagram showing a projected image of a single slit used in the conventional method. [Figure 9] FIG. 9 is a flowchart showing the process of generating a coded pattern according to an embodiment of the present invention. [Figure 10] FIG. 10 is a flowchart of a restoration according to an embodiment of the present invention. [Figure 11] FIG. 11 is a flowchart of the geometric constraint processing according to the embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing the results of the three-dimensional shape measurement in the example. [Figure 13] FIG. 13 is a graph showing the relationship between the number of shots and the restoration rate in the example. [Figure 14A] FIG. 14A shows an experiment conducted under conditions where overexposure occurs and the results thereof. [Figure 14B] FIG. 14B shows an experiment and the results thereof in a situation where interreflection occurs. [Figure 14C] FIG. 14C shows an experiment and the results for objects made of a variety of materials, including a specular object. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below are examples of how the present invention can be implemented, and the present invention is not limited to the specific configurations described below. When implementing the present invention, specific configurations corresponding to the embodiments may be appropriately adopted.

[0026] FIG. 1 is a diagram showing an application scene of a three-dimensional shape measurement system according to an embodiment of the present invention. The three-dimensional shape measurement system 100 includes a projector 10, a camera 20, and a three-dimensional shape measurement device 30. The projector 10 projects a projection image having a coded pattern (hereinafter also referred to as a "coded pattern image"). The camera 20 is installed so as to capture an image of a workpiece W, which is an object of three-dimensional shape measurement. The camera 20 captures the coded pattern image projected by the projector 10 and reflected by the moving workpiece W at a predetermined frame rate to generate a camera image.

[0027] The geometric relationship between the projector 10 and the camera 20, i.e., the relationship between their positions and orientations, is fixed during operation, and this geometric relationship is known in the three-dimensional shape measurement device 30. Furthermore, both the projector 10 and the camera 20 are connected to the three-dimensional shape measurement device 30 so as to be able to communicate with each other via wire or wirelessly. Note that while FIG. 1 shows an example in which the projector 10 and the camera 20 are configured as an integrated unit, the projector 10 and the camera 20 may also be configured as separate units.

[0028] The three-dimensional shape measuring device 30 measures the three-dimensional shape including position information of the workpiece W based on the coded pattern image projected by the projector 10 and the camera image captured by the camera 20. The three-dimensional shape measuring device 30 may be realized, for example, by a general-purpose computer executing the three-dimensional shape measuring program of this embodiment.

[0029] In the example of FIG. 1, multiple workpieces W are placed on a belt conveyor C, and the workpieces W move in a constant direction at a constant speed as the belt conveyor C rotates. The three-dimensional shape measurement device 30 is connected to an arm robot 40. The arm robot 40 is equipped with an arm having multiple joints, and its job is to grasp the moving workpiece W. The results of the three-dimensional shape measurement of the workpiece W obtained by the three-dimensional shape measurement device 30 are provided to the arm robot 40. The arm robot 40 operates based on the position information and three-dimensional shape information of the workpiece W. This allows the arm robot 40 to perform an appropriate task on the target workpiece W, for example, to appropriately pick up the workpiece W. Such a system is called a robot picking system.

[0030] In the following, the principle of three-dimensional shape measurement by the three-dimensional shape measurement system 100 of this embodiment will first be described, and then the configuration for performing such three-dimensional shape measurement will be described.

[0031] FIG. 2 is a diagram illustrating the principle of 3D shape measurement by a 3D shape measurement system according to an embodiment of the present invention. The 3D shape measurement system 100 of this embodiment measures the 3D shape of an object by using the known movement of the object as it is carried by a belt conveyor C as a geometric constraint called a motion constraint. The belt conveyor C is assumed to move in one direction at a constant speed, and the object placed on it and a point P on the object are also assumed to move in the same direction. The motion vector of this point P is called the object flow. A projected image 101 is projected by the projector 10, and a camera 20 captures a scene in which the object moves on the belt conveyor C, thereby obtaining a camera image 210.

[0032] In an environment where the movement of an object or the object flow can be precisely set, as in this embodiment, point P on the object moves outside the epipolar line in the camera image 201 captured by the camera 20 and the projected image 101 projected by the projector 10. In other words, a new constraint axis other than the epipolar constraint is given. The 3D shape measurement system 100 uses this constraint as a geometric constraint. This geometric constraint is called a motion constraint.

[0033] 3 is a diagram illustrating the processing steps of a decoding and hypothesis point determination algorithm using motion constraints according to an embodiment of the present invention. The measurement algorithm using motion constraints is outlined below. First, for an arbitrary point p0 in a camera image 201, a plurality of hypothesis points m1 to m4 that satisfy a geometric constraint condition in the depth direction on the camera line of sight L are set. Here, the camera line of sight L is a line connecting the focal point of the camera 20 and the arbitrary point P0 on the imaging plane of the camera.

[0034] Then, the multiple hypothesis points m1 to m4 are perspectively projected onto the camera image 201 and the projected image 101. This perspective projection is performed while each of the hypothesis points m1 to m4 is moved in the object flow direction F. At this time, the encoding pattern of the projected projected image is changed in synchronization with the movement. At this stage, hypothesis points that do not satisfy the geometric constraints are eliminated. Then, pairs of the multiple hypothesis points m1 to m4 are obtained, each pair consisting of a luminance change pattern (hereinafter also referred to as a "decoded signal") 202 perspectively projected onto the camera image and a luminance change pattern (hereinafter also referred to as a "coded signal") 102 perspectively projected onto the projector image. The correlation values ​​between the encoded signals and decoded signals of all these pairs are calculated, and the hypothesis point with the largest correlation value is designated as the restoration point. In the example of FIG. 3, the correlation of hypothesis point m2 is the largest, so hypothesis point m2 is designated as the restoration point.

[0035] 4 is a block diagram showing the functional configuration of a three-dimensional shape measurement system 100 according to an embodiment of the present invention. The three-dimensional shape measurement system 100 includes a projection unit 1, an imaging unit 2, an information processing unit 4, and a three-dimensional information output unit 5.

[0036] The projection unit 1 is composed of a projector 10. The photographing unit 2 is composed of a camera 20. The information processing unit 4 and the three-dimensional information output unit 5 may be provided in the three-dimensional shape measurement device 30, or only a portion thereof may be provided in the three-dimensional shape measurement device 30, and the other portion may be provided in the projector 10, the camera 20, or another device connected to the three-dimensional shape measurement device 30.

[0037] The projection unit 1 projects a coded pattern image based on the coded pattern. The projection unit 1 projects the coded pattern image onto an object moving on a belt conveyor C. Note that the projection unit 1 may be a device that can project any projection image, such as a projector 10, or may be a device that projects a projection image corresponding to a prepared plate, such as an overhead projector.

[0038] The photographing unit 2 photographs and obtains a camera image. The photographing unit 2 obtains a camera image sequence consisting of a plurality of consecutive camera images. In this embodiment, the photographing unit 2 detects only the luminance value of the incident light and generates a monochrome image as the camera image. Also, in this embodiment, the photographing unit 2 obtains the camera image using an image sensor with a resolution of 1024 x 768 pixels.

[0039] The information processing unit 4 includes a geometric calibration parameter acquisition unit 41, a motion parameter acquisition unit 42, a restoration unit 44, and a geometric constraint processing unit 45. The geometric calibration parameter acquisition unit 41 performs geometric calibration between the projection unit 1 and the image capture unit 2 and the object flow. When the relationship between the projection unit 1, the image capture unit 2, and the object flow is fixed, the geometric calibration parameters are constant. The motion parameter acquisition unit 42 acquires the object flow of an object undergoing a predetermined movement by estimation. The restoration unit 44 estimates depth (restored points) by decoding points that satisfy the motion constraints from among multiple depth hypothesis points using the camera image sequence, geometric calibration parameters, and motion parameters. When the restoration unit 44 estimates depth, the geometric constraint processing unit 45 excludes hypothesis points that do not satisfy the geometric constraints from the candidates.

[0040] The three-dimensional information output unit 5 outputs the set of restoration points obtained by the restoration unit 44 as the measurement result of the three-dimensional shape of the object.

[0041] 5 is a flowchart of a three-dimensional shape measurement method according to an embodiment of the present invention. First, the geometric calibration parameter acquisition unit 41 acquires pre-obtained geometric calibration parameters (step S41). Then, the motion parameter acquisition unit 42 acquires relative motion parameters between the projection unit 1 and the image capture unit 2 and the moving object, i.e., the object flow (step S42).

[0042] The projection unit 1 projects the projection image onto the object moving in accordance with the motion parameters acquired by the motion parameter acquisition unit 42 (step S43). The photographing unit 2 photographs N images of the object onto which the projection image is projected, and inputs the sequence of camera images to the information processing unit 4 as input data (step S44).

[0043] The reconstruction unit 44 generates M depth hypothesis points that satisfy the geometric constraint conditions defined by the geometric constraint processing unit 45 using the input data from the imaging unit 2, and adopts the depth hypothesis point that maximizes the score of the pair of the encoded signal 102 and the decoded signal 202 as the reconstruction point (step S45). Finally, the 3D information output unit 5 outputs the finally obtained 3D reconstruction result (step S46).

[0044] Fig. 6 is a flowchart of motion parameter estimation according to an embodiment of the present invention. Fig. 7 is a diagram showing an apparatus for estimating motion parameters according to an embodiment of the present invention. First, a calibration board 61 having a chess pattern is placed on an apparatus that moves in the same manner as the object to be measured (step S61). The calibration board is photographed at the same position as the object to be measured in photographing frame n, and then the board is moved to the next position and photographed, and this process is repeated (step S62).

[0045] The motion parameter acquisition unit 42 estimates motion parameters (R, T) n from frame n-1 to frame n using the sequence of camera images of the calibration board 61 (step S63). At this time, the motion parameter acquisition unit 42 estimates the motion parameters from the three-dimensional average motion flow of the corners of the chess pattern on the calibration board 61.

[0046] Specifically, for example, if a calibration board with a specific inclination is photographed seven times while being moved 10 mm at a time, focusing on a particular corner, six vectors are obtained from the difference between the 3D point of the corner in frame n and the 3D point of the corner in frame n-1, and the average of these vectors can be calculated to determine the average motion flow of that corner. This process is performed for all corners to determine the average motion flow for each, and then these are averaged to determine the overall average motion flow. The motion parameter acquisition unit 42 regards this overall average motion flow as the object flow, i.e., the motion parameters.

[0047] Note that even for devices with more complex movements, such as robot arms, the [Rt] of each corner can be calculated in a similar manner. Multiple calibrators can be used to obtain more dense corners, and areas without corners can be interpolated using linear interpolation or other methods to find the movement flow of surrounding corners.

[0048] 8A and 8B are diagrams showing examples of projected images according to an embodiment of the present invention. FIG. 8A shows an example of a projected image having a random dot pattern in which multiple dots are randomly arranged in a two-dimensional space, and FIG. 8B shows an example of a projected image having a random slit pattern in which multiple slits extending vertically are randomly arranged. In the example of FIG. 8A, the projected image has a pixel count of 1280 x 800 pixels, and the dot size is 10 x 10 pixels. In the example of FIG. 8B, the projected image has a pixel count of 1280 x 800 pixels, and the slit width is 10 pixels.

[0049] Figure 8C shows a projected image of a single slit used in conventional methods. The random dot pattern (Figure 8A) and random slit pattern (Figure 8B), which are coding patterns, have independent coding information embedded in each dot or slit, so measurements that are robust to optical phenomena can be expected. On the other hand, in the case of a single slit pattern (Figure 8C), if the projected light is reflected multiple times due to interreflection, it is expected to be difficult to identify direct reflections, making it difficult to measure the correct 3D points.

[0050] Compared to a single slit, a random dot pattern and a random slit pattern contain coded information, and therefore can be called coded patterns in comparison with a single slit. A random dot pattern is a pattern having coded information two-dimensionally in the vertical and horizontal directions, while a random slit pattern is a pattern having coded information only in either the vertical or horizontal direction (only in the horizontal direction in the case of Figure 8B).

[0051] 9 is a flowchart showing the process of generating the coding pattern shown in FIG. 8A or 8B. First, for each dot (in the case of random dots) or slit (in the case of random slits) of the coding pattern, a random number between 0 and 255 is generated based on an appropriate seed (step S81). In this embodiment, the random number is generated using a Mersenne Twister or M sequence. Then, for each dot / slit, if the generated random number is equal to or greater than a threshold value of 128, the pixel value of the dot / slit is set to 255, and if the generated random number is less than 128, the pixel value of the dot / slit is set to 0 (step S82).

[0052] 10 is a flowchart of restoration according to an embodiment of the present invention. The restoration unit 44 first captures a moving object onto which a coded pattern image is projected to acquire a camera image (step S101). Next, the restoration unit 44 sets M_p depth hypothesis points that satisfy two geometric constraints, the epipolar constraint and the depth constraint, at pixel p of the camera image (step S102).

[0053] The reconstruction unit 44 performs three-dimensional coordinate transformation of the hypothesis point m using the motion parameters (R, T)_n of each frame (step S103). The process of perspectively projecting the moved hypothesis point m_p onto the camera image and the projection image is repeated for N frames to obtain a pair of the encoded signal 102 and the decoded signal 202 (step S104) (see FIG. 3). The reconstruction unit 44 then obtains the correlation between the encoded signal 102 and the decoded signal 202 for the hypothesis point m_p using zero-mean normalized cross-correlation, and stores the obtained correlation value (step S105).

[0054] The restoration unit 44 determines whether the process of obtaining correlations at M_p hypothesis points has been completed (step S106), and if not completed (NO in step S106), returns to step S103 and repeats the process. If correlation values ​​have been obtained for M_p hypothesis points (YES in step S106), the restoration unit 44 adopts the hypothesis point m_p that has obtained the maximum correlation value among the M_p correlation values ​​as the restoration point for the pixel p (step S107). The restoration unit 44 performs the above process for all pixels of the camera image (step S108).

[0055] 11 is a flowchart of the geometric constraint processing according to the embodiment of the present invention. The geometric constraint processor 45 acquires a depth hypothesis point m_p at frame n, which is moved from the initial position (step S111). The geometric constraint processor 45 determines whether the depth value of the hypothesis point m_p is within the restoration range (depth constraint) (step S112). Here, while it is common to heuristically set the restoration range in advance based on the positional relationship between the measurement object and the camera 20 and projector 10, the depth constraint is a constraint that focuses on the depth direction.

[0056] If the depth value of the hypothesis point m_p is within the restoration range (YES in step S112), the geometric constraint processor 45 perspectively projects the hypothesis point m_p onto the camera image and the projection image (step S113), and determines whether the perspectively projected point is on the epipolar line (epipolar constraint) (step S114). The epipolar constraint is calculated from parameters (F matrix) obtained in advance by geometric calibration. The geometric constraint processor 45 determines whether the perspective projection point of the hypothesis point is on the epipolar line of both the camera image and the projection image.

[0057] The influence of calibration errors may be reduced by using a threshold that allows a certain distance from the epipolar line. For example, if the number of pixels of the camera 20 is 1024 × 768 pixels and the number of pixels of the projector 10 is 1280 × 800 pixels, the threshold th_C on the camera 20 side may be set to about 10 to 20 pixels, and the error on the projector 10 side may be allowed based on the ratio of the pixel size to the threshold (threshold th_P = th_C × (1280 / 1024)).

[0058] If the perspectively projected point is on the epipolar line (YES in step S114), that is, if both the depth constraint and the epipolar constraint are satisfied, the geometric constraint processor 45 determines that the depth hypothesis point m_p is a hypothesis point that satisfies the geometric constraint (step S115). If either the depth constraint or the epipolar constraint is not satisfied (NO in step S112 or NO in step S114), the geometric constraint processor 45 determines that the depth hypothesis point m_p is a hypothesis point that does not satisfy the geometric constraint (step S116).

[0059] (Example) An example of three-dimensional shape measurement will be described below. Using the three-dimensional measurement system 100 shown in FIG. 2, a Lambertian model of the Stanford Bunny (10×10×10 cm) was used as the object to be measured. 3We conducted an experiment to measure the 3D fur texture of a moving bunny (approximately 100mm in size) by taking N images while moving it. Using the camera images and projected images, we performed 3D shape measurement (shape reconstruction) of a bunny moving with the known movement (object flow) described above, using a measurement algorithm with motion constraints (see Figure 10).

[0060] Fig. 12 shows the results of 3D shape measurement in the example. In Fig. 12, the top row shows the results when a random dot pattern was used, the middle row shows the results when a random slit pattern was used, and the bottom row shows the results when a single slit pattern was used. Also, in Fig. 12, the left side of each row shows the results when the number of images taken was N = 9 (image taken every 6 mm of movement), and the right side of each row shows the results when the number of images taken was N = 54 (image taken every 1 mm of movement).

[0061] As shown in Figure 12, for a Lambertian object, if enough images are taken (far right), the single slit pattern can capture the correct shape. However, the single slit pattern produces more noise than the random dot and random slit patterns.

[0062] Fig. 13 is a graph showing the relationship between the number of captured images and the restoration rate in the embodiment. Fig. 13 shows the restoration rates for the number of captured images N = 27, 18, and 9 for the random dot pattern, random slit pattern, and single slit, respectively. Note that a large number of captured images N means that the object's movement speed is slow and / or the frame rate of the camera 20 is low, and a small number of captured images N means the opposite. Therefore, the fact that a high restoration rate can be maintained even when the number of captured images N is small means that the object's movement speed can be increased and / or a camera 20 with a low frame rate will suffice.

[0063] As shown in Fig. 13, in the case of the random dot pattern and the random slit pattern, a higher restoration rate (high density) can be achieved compared to the single slit, and even when the number of images taken N is reduced, the decrease in restoration rate (density) is kept small compared to the single slit case. This is also clear when comparing the left and right columns in Fig. 12.

[0064] Furthermore, we conducted experiments to investigate robustness against optical phenomena by using objects with various reflectance properties. Figure 14A shows the experiment and its results in a situation where overexposure occurs, Figure 14B shows the experiment and its results in a situation where interreflection occurs, and Figure 14C shows the experiment and its results for objects made of multiple materials, including specular objects.

[0065] As shown in Figure 14A, when halation (blown-out highlights) occurs in the camera image, it was confirmed that the effects of missing data due to whiteout can be suppressed for the random dot pattern and random slit pattern. Furthermore, as shown in Figure 14B, when interreflection occurs, an incorrect shape is obtained in the area indicated by the arrow for the random slit pattern. Similarly, an incorrect shape is obtained in the area indicated by the arrow for the single slit. On the other hand, the random dot pattern, which is considered to be robust against interreflection, was able to recover the shape well overall.

[0066] In Figure 14C, the top row shows the reconstruction results as seen from the front, and the bottom row shows the reconstruction results as seen from above. In the example in Figure 14, stainless steel, polystyrene foam, and wooden balls were placed in that order and 3D shape measurement was performed. As shown in Figure 14C, when a random slit pattern was used, an incorrect shape was measured at the area indicated by the arrow due to interreflection, and even with a single slit, shape measurement was not correct due to the influence of interreflection between the stainless steel and polystyrene foam. With the random dot pattern, the influence of interreflection was suppressed overall and shape reconstruction was possible.

[0067] The following findings were obtained from the above experiments. Namely, the random slit pattern has few boundaries between lines and is less affected by adjacent lines, and tends to provide particularly good shape reconstruction in scenes where optical phenomena are not severe. However, it is susceptible to the effects of interreflection outside the epipolar line, and in scenes where interreflection occurs, it tends to result in erroneous measurements and the creation of somewhat large clusters of noise.

[0068] Furthermore, although random dot patterns tend to be easily affected by adjacent dots during coordinate decoding, making it difficult to restore points, they were found to have the advantage of being highly robust against interreflection, making it difficult to restore noise points.In both the random slit pattern and the random dot pattern coding patterns, the high robustness against optical phenomena is thought to be achieved by suppressing the influence of the inside and outside of the epipolar line, as the random code is arranged as slits or dots.

[0069] As described above, in this embodiment, a projected image having a random pattern is projected from projector 10 onto an object moving along a known object flow, and the object is photographed by camera 20. A plurality of depth hypothesis points are set at arbitrary points on the camera's line of sight in the camera image, and the hypothesis points are perspectively projected onto the camera image and the projection image while moving along the object flow. The luminance change of the hypothesis point in the camera image is compared with the time-series change in the projection image, and the hypothesis point with the highest correlation therebetween is determined as the restoration point. This makes it possible to measure the three-dimensional shape of the object without stopping the movement of the object, even without using high-speed (high frame rate) projector 10 and camera 20.

[0070] In the above embodiment, it is assumed that the object to be measured has reflection characteristics close to Lambertian, but this embodiment may also be applied to cases where the object is not a Lambertian reflector. When measuring in a relatively stationary state, areas where restoration is missing occur due to the influence of specular reflected light, but by using this embodiment to obtain information about the object while it is moving, it is possible to reduce this loss.

[0071] In the above embodiment, the coding pattern in the projection image is kept constant and not changed in order to obtain multiple camera images. However, instead, projection images having different coding patterns may be projected for the multiple camera images. In this case, the three-dimensional shape measurement system 100 may include a synchronization control unit that synchronizes the rate at which the coding pattern of the projection image by the projection unit 1 changes with the frame rate of imaging by the imaging unit 2. In this case, the three-dimensional shape measurement system 100 may also include a coding pattern generation unit for generating a changing coding pattern.

[0072] Furthermore, it is desirable that the dots of the monochrome coding pattern generated by the coding pattern generation unit 43 described above are larger, as this can absorb errors in geometric calibration and motion estimation. Furthermore, when the restoration unit 44 sets hypothesis points and moves the hypothesis points along the object flow, there are cases where the hypothesis points correspond to boundaries (boundary pixels) between dots in the coding pattern.

[0073] In this case, such a hypothesis point will have a pixel value that is intermediate between white and black. Therefore, hypothesis points that often become boundary pixels may be excluded from candidates for restoration points. For example, when a hypothesis point is moved over 50 frames, if 20 or more of the hypothesis points (40%) become boundary pixels, such a hypothesis point may be excluded from candidates for restoration points.

[0074] Furthermore, although the above embodiment uses one projector 10 and one camera 20, multiple projectors and cameras may be used. For example, even if one projector and two cameras or three projectors and three cameras are combined, if each projector generates its own unique coded pattern image sequence and the ZNCC of the coded signal generated by each projector is relatively low, coded information can be separated.

[0075] In the above embodiment, the known movement is realized using a belt conveyor, i.e., a horizontally moving slide stage. However, in addition to or instead of this, the known movement may be realized using a device with a known operation, such as a robot arm or an AGV. Because robot arms and AGVs are parameter-controlled by humans, converting their control signals into rotation and translation information makes it possible to perform calculations equivalent to the translational movement of a hypothetical point using a slide stage. Even with devices such as robot arms that perform complex movements, it is possible to determine the [Rt] of each corner during calibration. Multiple calibrators may be used to more closely locate corners, or areas without corners may be interpolated using linear interpolation of the flow of surrounding corners. [Explanation of symbols]

[0076] 1 Projection section 2. Filming Department 4. Information Processing Section 41 Geometric calibration parameter acquisition unit 42 Motion parameter acquisition unit 44 Restoration Section 45 Geometric constraint processing unit 5. 3D information output section 10 Projector 20 Camera 30 3D shape measurement device 40 Arm Robot 100 3D shape measurement system 101 Projected Images 102 Brightness change pattern of the hypothetical point perspectively projected onto the projection image (encoded signal) 201 camera images 202 Brightness change pattern of the hypothetical point perspectively projected onto the camera image (decoded signal) double work

Claims

1. A three-dimensional shape measurement system for measuring a three-dimensional shape of an object that moves in a known manner, comprising: a projection unit that projects a projection image having a coding pattern onto the object having the known movement; an imaging unit that continuously captures images of the object that is moving in a known manner and onto which the one projection image is projected, to generate a plurality of camera images; a restoration unit that sets a plurality of hypothesis points on a camera line of sight for an arbitrary point of the camera image, and determines, from among the plurality of hypothesis points, a hypothesis point that satisfies a motion constraint based on the known motion as a restoration point for the arbitrary point; A three-dimensional shape measurement system equipped with the above.

2. The three-dimensional shape measurement system according to claim 1 , wherein the coded pattern is a random slit pattern in which a plurality of slits are randomly arranged.

3. The three-dimensional shape measurement system according to claim 2 , wherein the extension direction of the slit is a direction perpendicular to the known movement.

4. 2. The three-dimensional shape measurement system according to claim 1, wherein the coded pattern is a random dot pattern in which a plurality of dots are randomly arranged two-dimensionally.

5. the restoration unit moves the plurality of hypothesis points by a motion parameter representing the known motion, 2. The three-dimensional shape measurement system according to claim 1, wherein the hypothesis point at which a correlation between a set of luminance change patterns obtained by perspective projecting the moved hypothesis points onto the camera image and the projection image respectively is maximized is determined to satisfy the motion constraint, and the hypothesis point is determined to be the restoration point.

6. 3. The three-dimensional shape measurement system according to claim 1, further comprising a geometric constraint processing unit that, when the initial hypothesis point set by the restoration unit and the moved hypothesis point do not satisfy an epipolar constraint and / or a depth constraint, excludes the hypothesis point from candidates for the restoration point.

7. 6. The three-dimensional shape measurement system according to claim 5, further comprising a motion parameter acquisition unit that uses a calibration pattern instead of the object to determine a movement flow of restoration points of points on the calibration pattern obtained by the projection unit, the photographing unit, and the restoration unit, thereby determining the motion parameters.

8. A three-dimensional shape measurement method for measuring a three-dimensional shape of an object having a known movement, comprising: a projection step of projecting a projection image having a coding pattern onto the object having the known movement; an imaging step of continuously imaging the object that is moving in a known manner and onto which the one projection image is projected, to generate a plurality of camera images; a restoration step of setting a plurality of hypothesis points on a camera line of sight for an arbitrary point of the camera image, and determining, among the plurality of hypothesis points, a hypothesis point that satisfies a motion constraint based on the known motion as a restoration point for the arbitrary point; A three-dimensional shape measurement method, comprising:

9. A three-dimensional shape measurement program for measuring the three-dimensional shape of a known moving object, which is executed by an information processing device connected to a projector and a camera, causing the projector to project a projected image having a coding pattern onto the object having the known movement; causing the camera to continuously capture images of the object with known movement onto which the one projection image is projected to generate a plurality of camera images; setting a plurality of hypothesis points on a camera line of sight for an arbitrary point of the camera image, and determining, among the plurality of hypothesis points, a hypothesis point that satisfies a motion constraint based on the known motion as a restoration point for the arbitrary point; 3D shape measurement program.

Images