Measuring device, camera device, measuring system, control method and storage medium

Abstract

A measuring device comprises a projection device configured to project a predetermined pattern on a subject; and a camera system configured to capture a group of images from at least two different viewpoints, wherein the distance between the viewpoints is shorter than the distance between the projection device and the viewpoints, the measuring device further comprising: a relative position calculation unit configured to obtain a relative position of the projection device relative to at least one viewpoint based on a pattern image position on the image group and a positional relationship between the viewpoints, wherein distance information related to the subject is obtained based on the relative position and the pattern image position on the image at the viewpoint.

Description

Technical Field

[0001] The present invention relates to a measuring device, an imaging device, a measuring system, a control method, and a program, and in particular, to a measuring technique using a plurality of distance measuring methods including a stereo method in a combined manner. Background Art

[0002] So far, a technique for obtaining distance information and shape information of a subject based on an image obtained by shooting using the principle of triangulation (so-called stereo method) has been proposed. For example, a passive stereo method is known for measuring the shape of a subject by using a parallax image pair obtained by shooting a subject with two camera devices located at different positions, using the principle of triangulation. In addition, a pattern projection stereo method is known for measuring the shape of a subject with a small amount of texture or measuring the shape of a subject in a dark place. In the pattern projection stereo method, a predetermined pattern is projected onto the subject by a projection device, an image is shot by a camera device located at a position different from the position of the projection device, and based on the positional relationship between the projection device and the camera device and the positional relationship of the pattern on the image obtained by shooting, the distance information and shape information related to the subject are obtained using the principle of triangulation. In addition, in order to improve accuracy, etc., a hybrid method of using the pattern projection stereo method and the passive stereo method in a combined manner is proposed (patent document 1).

[0003] Reference List

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2002-286415 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] In Patent Document 1, a hybrid method using a pattern projection stereo method and a passive stereo method in combination is proposed, but a specific structure and method are not disclosed.

[0008] Since stereo method uses the principle of triangulation, it is important that the positional relationship between the two elements causing the parallax is predetermined (known) and stable in order to obtain high measurement accuracy. Here, in passive stereo method, the two elements are two camera devices, and in pattern projection stereo method, the two elements are a projection device and a camera device. Therefore, when passive stereo method and pattern projection stereo method are used in combination, it is important that the positional relationship between the two camera devices and the positional relationship between one camera device and a projection device are known and stable in order to obtain high measurement accuracy.

[0009] Furthermore, due to the triangulation principle, in order to obtain high measurement accuracy, it is advantageous that the distance between the two elements for causing the parallax (base line length) is longer.

[0010] However, in order to make the positional relationship between the two elements known and stable under the temperature environment and the presence of vibration, the two elements need to be firmly fixed, and there is a problem that the structure of the measurement system becomes complicated and the size of the measurement system increases. In addition, if the baseline length, which is the distance between the two elements, is increased in order to perform high-precision measurement, the mechanical stability is reduced, and thus there is a problem that the size of the mechanism for holding the two elements is further increased.

[0011] On the other hand, if the positional relationship between the projection device and the camera device is fixed, the adjustment range and direction of the projection position are limited, because, for example, the pattern needs to be projected according to the shape of the subject so as not to produce a shadow, etc., thereby causing a problem of reduced usability.

[0012] In addition, by projecting a pattern using a projection device to form a texture on the subject, it is possible to measure the shape of a subject with a small amount of texture or to measure a subject in a dark place using a passive stereo method. However, in this case, the pattern projection is only intended to improve the passive stereo method, and this structure is different from the structure using the pattern projection stereo method in a combined manner. Therefore, there is a problem of low measurement stability and accuracy compared to the case of the hybrid type.

[0013] The present invention has been made to solve at least one of the problems of the known techniques, and aims to provide a measuring device, an imaging device, a measuring system, a control method, and a program for achieving good measurement accuracy while suppressing an increase in device scale and a decrease in usability.

[0014] Solutions for solving problems

[0015] A measuring device comprises: a projection device configured to project a predetermined pattern on a subject; and a camera system configured to capture a group of images from at least two different viewpoints, wherein the distance between the viewpoints is shorter than the distance between the projection device and the viewpoints, and the measuring device further comprises: a relative position calculation unit configured to obtain a relative position of the projection device relative to at least one of the viewpoints based on a pattern image position on the image group and a positional relationship between the viewpoints, wherein distance information related to the subject is obtained based on the relative position and the pattern image position on the image at the viewpoint.

[0016] Advantageous Effects of the Invention

[0017] According to the present invention, good measurement accuracy can be achieved while suppressing an increase in equipment scale and a decrease in usability.

[0018] Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. Note that the same reference numerals in the accompanying drawings represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a block diagram showing a functional structure of a measurement system according to an embodiment of the present invention.

[0021] Figure 2A : is a flowchart showing the distance measurement process performed by the measurement device 150 according to the first embodiment.

[0022] Figure 2B is a diagram for describing the pattern projection mode in the first embodiment.

[0023] Figure 3A : is a flowchart showing a modification of the distance measurement process performed by the measurement device 150 according to the first embodiment.

[0024] Figure 3B : is a flowchart showing another modification of the distance measurement process performed by the measurement device 150 according to the first embodiment.

[0025] Figure 4 is a diagram showing a method of calculating the position and posture of the projection device 110 according to the first embodiment.

[0026] Figure 5A is a block diagram showing the functional structure of a measurement system according to the second embodiment.

[0027] Figure 5Bis a diagram for describing a detailed configuration of an imaging system 120 according to the second embodiment.

[0028] Fig. 6A : is a flowchart showing a distance measurement process performed by the measurement device 150 according to the third embodiment.

[0029] Figure 6B : is a diagram showing a captured image that can be obtained in a state where pattern projection according to the third embodiment is performed.

[0030] Fig. 7A : is a flowchart showing a modification of the distance measurement process performed by the measurement device 150 according to the third embodiment.

[0031] Figure 7B : is a flowchart showing another variation of the distance measurement process performed by the measurement device 150 according to the third embodiment.

[0032] Fig. 8A : is a flowchart showing a distance measurement process performed by the measurement device 150 according to the fourth embodiment.

[0033] Figure 8B : is a diagram showing a captured image that can be obtained in a state where pattern projection according to the fourth embodiment is performed. DETAILED DESCRIPTION

[0034] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Although a plurality of features are described in the embodiments, the present invention is not limited to requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar structures, and redundant descriptions thereof are omitted.

[0035] One embodiment described below describes an example in which the present invention is applied to a measuring system, as an example of a measuring system, which includes two imaging devices (cameras) and a projection device, and a measuring device connected to these devices and capable of measuring the distance between the imaging devices and the subject. However, the present invention can be applied to any device capable of performing distance measurement in the depth direction using a passive stereo method in a state of performing pattern projection.

[0036] [First embodiment]

[0037] exist Figure 1 An exemplary structure of the measurement system 100 in the present invention is shown in FIG.

[0038] exist Figure 1In the embodiment, the measuring system 100 is composed of a projection device 110 that projects a pattern on a subject, an imaging system 120 that acquires a parallax image based on a light beam from the subject, and a measuring device 150 .

[0039] The projection device 110 is composed of a projection optical system 111, a pattern unit 112 including a spatial modulator or a reticle, and an illumination unit such as an LED, etc. Reference numeral 113 indicates an exit pupil of the projection optical system 111, and reference numeral 114 indicates a projection axis representing a projection direction of the projection device 110. The projection device 110 includes a control unit (not shown) including a ROM in which a control program of a block of the projection device 110 is recorded and a RAM used as a development area of ​​the control program, and the control unit projects a pattern (light) on a subject by controlling the pattern unit 112.

[0040] The imaging system 120 is composed of a first camera 130 and a second camera 140, and captures a group of images from two different viewpoints. The first camera 130 includes a first imaging optical system 131 and a first image sensor 132 such as a CCD sensor or a CMOS sensor, and captures an image with a first entrance pupil 133 of the first imaging optical system 131 as a viewpoint and a first imaging axis 134 as an imaging direction. The second camera 140 includes a second imaging optical system 141 and a second image sensor 142 such as a CCD sensor or a CMOS sensor, and captures an image with a second entrance pupil 143 of the second imaging optical system 141 as a viewpoint and a second imaging axis 144 as an imaging direction. Note that such a structure makes the distance between the first entrance pupil 133 and the second entrance pupil 143 shorter than the distance between the exit pupil 113 of the projection device and the first entrance pupil 133 of the imaging system 120. In addition, the first camera 130 and the second camera 140 each include constituent elements included in a general digital camera, such as a control unit composed of a CPU, ROM and RAM, and an image processing unit for performing image processing such as development and correction of image signals output from an image sensor and generating digital data of the captured image.

[0041] The measuring device 150 includes a control unit 151 composed of a CPU, a memory 152, a relative position calculation unit 154, and a distance / shape calculation unit 153. The control unit 151 controls the operation of the block by reading out the operation program of the block stored in the non-volatile part of the memory 152 (hereinafter referred to as ROM for convenience), developing the operation program in other areas of the memory 152, and executing the program. In addition, the measuring device 150 is connected to the camera system 120 and the projection device 110 as external devices, and enables data transmission and reception between the devices.

[0042] Next, use Figure 2A The flow of calculating distance and shape performed by the measuring system 100 is described. As a result of the control unit 151 of the measuring device 150 executing a corresponding control program, each processing step of the present flow is performed by the control unit 151 or a block operating according to the control of the control unit 151.

[0043] In step S1501, the control unit 151 instructs the projection device 110 to project a pattern via the communication unit, and instructs the camera system 120 to capture an image in a state where the pattern is projected, and acquires a parallax image pair obtained by capturing from the camera system 120. Figure 2B As shown, the projection device 110 projects the pattern 200 on the object in the direction of the projection axis 114 through the pattern unit 112 and the projection optical system 111 when receiving an instruction from the measuring device 150. The camera system 120 uses the first camera 130 and the second camera 140 to capture the parallax image of the object and the pattern 200 when receiving an instruction from the measuring device 150.

[0044] In step S1502 , the distance / shape calculation unit 153 calculates primary distance information about the subject using a known method of passive stereo based on the position of the subject on the captured parallax image and the position of the pattern image and the positional relationship between viewpoints.

[0045] Next, in step S1503, the relative position calculation unit 154 calculates the position of the exit pupil 113 and the projection axis 114, which are the position and posture of the projection device 110 relative to the first entrance pupil 133 of the imaging system 120, based on the primary distance information of the object and the position of the pattern image on the parallax image. Note that in the present embodiment, a description will be given of the relative position and posture based on the position and posture of the first camera 130 as the position and posture of the projection device 110. However, the relative position and posture based on the position and posture of the second camera 140 may be adopted.

[0046] Next, in step S1504, the distance / shape calculation unit 153 calculates secondary distance information of the subject using a known method of pattern projection stereo based on the calculated position and posture of the projection device 110, the position and posture of the camera system 120, and the position of the pattern image on the parallax image.

[0047] Then, in step S1505, the control unit 151 stores the calculated secondary distance information in a storage unit (not shown), or outputs it to an external device via a recording unit or a communication unit, etc. Here, the distance (baseline length) between the exit pupil 113 of the projection device 110 and the first entrance pupil 133 of the camera system 120 is longer than the distance (baseline length) between the first entrance pupil 133 and the second entrance pupil 143 of the camera system 120. Therefore, the distance accuracy of the secondary distance information obtained by the pattern projection stereo method performed by the projection device 110 is higher than the distance accuracy of the primary distance information obtained by the passive stereo method performed by the camera system 120. Therefore, the first camera 130 and the second camera 140 of the camera system 120 only need to maintain a positional relationship in which the baseline length is shorter than the baseline length required to obtain the desired measurement accuracy, thereby obtaining a higher distance measurement accuracy with a simpler structure.

[0048] In addition, if Figure 3A As shown, except Figure 2A In addition to the processing performed in step S1506, the control unit 151 may also perform the processing of step S1506, in which the distance information of the subject is obtained by integrating the primary distance information obtained by the passive stereo method and the secondary distance information obtained by the pattern projection stereo method. Note that Figure 3A Each process in the flow is also executed by the control unit 151 of the measuring device 150 or by a block that operates according to the control of the control unit 151. Depending on the relationship between the position of the projection device 110 and the shape of the subject, there is a case where an area where a pattern is not projected due to the shadow of the subject appears. In this case, the primary distance information can be acquired using a passive stereo method using the subject pattern as a clue, whereby the distance information without loss can be obtained by integrating the primary distance information with the secondary distance information.

[0049] Next, use Figure 4 A method of calculating the position of the exit pupil 113 and the projection axis 114 indicating the position and posture of the projection device 110 from the primary distance information of the subject and the position of the pattern image on the parallax image is described, which is executed by the control unit 151 in step S1503 . Figure 4A case where a distance is measured by projecting a pattern on the object 300 is shown. A pattern is projected in the direction of a projection axis 114 from an exit pupil 113 of a projection device 110, and a parallax image related to a first entrance pupil 133 of a first camera 130 and a second entrance pupil 143 of a second camera 140 is acquired. First, distance information is calculated once for any two points (pattern 201, pattern 202) on a pattern formed on the surface of the object 300, and relative position coordinates of the position of the pattern relative to the first entrance pupil 133 of the first camera 130 are obtained. The relative position coordinates relative to the first entrance pupil 133 are position coordinates in a rectangular coordinate system that takes, for example, the first entrance pupil 133 as an origin, a straight line connecting the first entrance pupil 133 and the second entrance pupil 143 as one axis, and sets another axis in a direction perpendicular to the axis. Next, when the projection angle between the pattern 201 and the pattern 202 is represented as θ1, a circle 310 passing through two points of the pattern 201 and the pattern 202 is obtained, where θ1 is a circumferential angle. Here, the exit pupil 113 of the projection device exists at a certain position on the circumference of the circle 310. Note that the projection angle θ1 is known information about the projection device 110, and is determined, for example, based on the geometric relationship between the pattern unit 112 and the projection optical system 111. Next, the primary distance information related to another point (pattern 203) on the pattern formed on the surface of the object 300 is calculated, and the relative position coordinates of the position of the pattern relative to the first incident pupil 133 of the first camera 130 are obtained. When the projection angle between the pattern 202 and the pattern 203 is represented as θ2, a circle 311 passing through two points of the pattern 202 and the pattern 203 is obtained, where θ2 is a circumferential angle. Here, the exit pupil 113 of the projection device exists at a certain position on the circumference of the circle 311. That is, the intersection of the circle 310 and the circle 311 that is not the pattern 202 indicates the position of the exit pupil 113. In the present embodiment, the pattern on the projection axis is selected as the pattern 202. Here, the straight line including the line segment connecting the position of the exit pupil 113 and the position of the pattern 202 indicates the projection axis 114. As described above, the position of the exit pupil 113 and the projection axis 114 indicating the position and posture of the projection device 110 can be calculated based on the primary distance information of the object and the position of the pattern image on the parallax image.

[0050] Note that although in the present embodiment, relative position information of three points on the surface of the object is used, information related to four points or more than four points may be used to obtain the position of the exit pupil 113 and the projection axis 114 with higher accuracy using averaging processing or the like. In addition, if there is prior information related to the position coordinates of the exit pupil 113 and the direction of the projection axis 114, the position of the exit pupil 113 and the projection axis 114 may be obtained using the relative position information of the two points. For example, if the inclination of the projection axis 114 relative to the coordinate system with the first incident pupil 133 as the origin is known, the intersection between the circle 310 obtained based on the relative position information of the two points and the projection axis 114 at the position passing through the pattern 202 indicates the position of the exit pupil 113. Note that using a method such as Figure 4 The method of obtaining the position of the exit pupil 113 of the projection device 110 and the projection axis 114 based on the geometric relationship shown in the embodiment is not limited to the processing steps shown in the present embodiment. The method only needs to be a method of obtaining the relative position of the projection device relative to at least one viewpoint using the pattern image position on the parallax image acquired by the camera system 120 and the positional relationship between the viewpoints as input information, and may also be a method that does not use intermediate data such as primary distance information.

[0051] In this embodiment, use Figure 4 A method of obtaining the position and orientation of the projection device 110 is described, wherein a diagram (construction) is drawn in a two-dimensional plane including a line segment connecting the first entrance pupil 133 and the second entrance pupil 143 of the camera system 120. If the position of the exit pupil 113 of the projection device 110 does not exist in this two-dimensional plane, a diagram (construction) may also be drawn in a two-dimensional plane including the first entrance pupil 133 of the first camera 130 and connecting the first entrance pupil 133 and the second entrance pupil 143 of the camera system 120. Figure 4 The position and posture of the projection device 110 are obtained using geometric relationships on planes orthogonal to the plane shown in . The three-dimensional position and posture can be obtained by combining the positions and postures obtained on these planes. Note that preferably, the projection device 110 is configured at a position on the extension line of the straight line connecting the first incident pupil 133 and the second incident pupil 143 of the camera system 120. Accordingly, it is only necessary to obtain the position and posture of the projection device 110 on the two-dimensional plane, and the amount of calculation can be reduced, thereby achieving high speed and suppressing power consumption.

[0052] According to the structure as described above, in the present embodiment, it is only necessary to maintain a positional relationship in which the base line length is shorter than the base line length required to obtain desired measurement accuracy, whereby high distance measurement accuracy can be stably obtained with a simpler structure.

[0053] In addition, the position and posture of the projection device 110 can be obtained, so the position and posture of the projection device 110 can be changed when necessary, instead of fixing the position and posture of the projection device 110 in a predetermined state. Therefore, the direction in which the pattern is projected and the position of the pattern can be determined so that there will be no obstruction due to the shape of the subject, so the conditions under which the subject can be measured can be greatly expanded. Note that it is not necessary to obtain the position and posture of the projection device 110 every time measurement is performed. For example, it may be a structure such that the position and posture of the projection device 110 are obtained when the position and posture of the projection device 110 are actively changed, or when a change in the positional relationship caused by a change in ambient temperature or a disturbance such as vibration is taken into account.

[0054] Note that the pattern to be projected is preferably a stripe pattern in which line segments extending in a direction perpendicular to the straight line connecting the first entrance pupil 133 and the second entrance pupil 143 of the camera system 120 are arranged approximately periodically in the direction of the straight line (the direction in which parallax appears in the parallax image pair). According to such a structure, a parallax image with high contrast in the parallax direction of the camera system 120 can be obtained, and the measurement accuracy of the primary distance information by the passive stereo method can be improved. In the camera system 120, such good pattern projection can also be achieved in a mode in which a pattern is dynamically configured based on a captured image of the test pattern after the test pattern projection is performed.

[0055] In addition, it is desirable that at least one line segment (stripes) of the stripe pattern has a color or shape different from the color or shape of the other line segments. Accordingly, the error when comparing the parallax images can be reduced, and the measurement error can be reduced. In particular, the color or shape of the pattern on the projection axis 114 is changed relative to the color or shape of the other line segments, and as a result, the calculation of the projection axis 114 can be simplified. In addition, preferably, information related to the pattern 200 (positional relationship between patterns) to be projected in order to measure the distance information once is also stored in advance on the camera system side, so that when the camera system 120 recognizes the projected pattern image, the relative positional relationship between the patterns can be understood. Accordingly, the camera system 120 easily estimates the positional relationship of the subject at the position where the pattern is projected in the pupil division direction based on the positional relationship between the patterns (for example, the positional relationship between the stripes in the stripe pattern).

[0056] In addition, a structure may be adopted in which the pattern used when obtaining the relative position of the projection device using the passive stereo method is different from the pattern used when obtaining the distance information using the pattern projection stereo method. Figure 2A In addition to the processing shown in FIG. 1 , the measurement system 100 may use Figure 3B The measurements were performed according to the procedure shown in . Note that Figure 3BEach process in the process is also performed by the control unit 151 of the measuring device 150 or by a block that operates according to the control of the control unit 151. After step S1503, in step S1507, the projection device 110 projects a pattern such as random points or grades commonly used in the pattern projection stereo method, shoots the subject to which the pattern is projected through the camera system 120, and obtains a parallax image pair. According to such a structure, a pattern suitable for each method can be used, and position calculation and distance measurement with higher accuracy can be performed. In addition, by changing the pattern used in the pattern projection stereo method to a pattern suitable for the shape of the subject using the primary distance information (such as changing the fineness of the pattern) and projecting the changed pattern, the distance information can be obtained with higher accuracy.

[0057] [Second embodiment]

[0058] Next, use Figure 5A and 5B The second embodiment is described. Figure 5A As shown, the second embodiment is different from the first embodiment in that the camera system 120 is composed of a camera 400. The processing flow includes Figure 2A The example shown in , and the processing flow is the same as that of the first embodiment.

[0059] The camera 400 is composed of an imaging optical system 401 and an image sensor 402. In the image sensor 402, one pixel includes two photoelectric conversion units. Figure 5B As shown, each pixel of the image sensor 402 is composed of a microlens 403, a first photoelectric conversion unit 404, and a second photoelectric conversion unit 405. Via the microlens 403, the photoelectric conversion unit and the pupil of the imaging optical system 401 are in an optical conjugate relationship, and parallax images from different viewpoints are acquired by dividing the pupil of the imaging optical system. The image obtained using the image signal from the first photoelectric conversion unit 404 corresponds to an image with the second entrance pupil 143 in the first embodiment as a viewpoint, and the second entrance pupil 143 corresponds to a part of the pupil area of ​​the imaging optical system 401. In addition, the image obtained using the image signal from the second photoelectric conversion unit 405 of the pixel is an image with the first entrance pupil 133 in the first embodiment as a viewpoint, and the first entrance pupil 133 corresponds to another part of the pupil area of ​​the imaging optical system 401. In addition, the image obtained using the signal obtained by adding the image signal from the first photoelectric conversion unit 404 of the pixel and the image signal from the second photoelectric conversion unit 405 of the pixel is an image with the entrance pupil 406 as a viewpoint.

[0060] As described above, in this embodiment, unlike the first embodiment using multiple cameras, a camera optical system 401 and an image sensor 402 can be used to acquire a parallax image pair, whereby the camera system can be configured to be mechanically stable and have a short baseline length, and stable and high distance measurement accuracy can be obtained with a simpler structure.

[0061] In addition, a normal image obtained by shooting without projecting a pattern is also obtained, so measurement values ​​with the same viewpoint as the normal image can be obtained, and the measurement values ​​are added to the image as intermediate information, or the measurement values ​​are used for recognition, resulting in a camera device with higher recognition accuracy.

[0062] [Third embodiment]

[0063] Next, use Fig. 6A and 6B The third embodiment is described. Note that Fig. 6A Each process in the flow is also executed by the control unit 151 of the measuring device 150 or by a block operating according to the control of the control unit 151. Fig. 6A The processing flow is shown, and in step S1511, a pattern is projected and photographed. Figure 6B An example of an image having a viewpoint as the first entrance pupil 133 is shown, where the image is obtained using Figure 5A The measurement system 100 shown in FIG. 1 is obtained by projecting a pattern by a projection device 110 and photographing by an imaging system 120. The pattern to be projected is preferably a stripe pattern in which line segments extending in a direction perpendicular to a straight line connecting a first entrance pupil 133 and a second entrance pupil 143 of the imaging system 120 are arranged approximately periodically in the straight line direction. The subject is a planar surface and a hemispherical object located on the planar surface.

[0064] In step S1512 , the distance / shape calculation unit 153 calculates primary distance information of the object from the captured parallax images using the passive stereo method.

[0065] Next, in step S1513, the distance / shape calculation unit 153 generates surface normal information based on the fringe interval of the fringe pattern image in one parallax image. The fringe interval of the fringe pattern image indicates the inclination of the surface in the fringe period direction, and when the surface faces the observation direction, the fringe interval is the same as the fringe interval of the projection pattern, and when the surface normal is inclined, the fringe interval decreases. Figure 6B As shown, for example, the stripes in the region 210 which is a planar surface are straight lines and are equally spaced. On the other hand, the stripes in the region 211 which is a curved surface are curved lines, and the stripe intervals vary depending on the position.

[0066] In step S1513, the distance / shape calculation unit 153 obtains the local fringe interval at the position on the image, and based on this, the surface normal information at the local position of the object can be obtained. In addition, in step S1514, the distance / shape calculation unit 153 generates high-precision distance and shape information by integrating the surface normal information with the primary distance information.

[0067] Then, in step S1515 , the control unit 151 stores the calculated high-precision distance and shape information in a storage unit (not shown), or outputs it to an external device via a recording unit or a communication unit or the like.

[0068] The primary distance information obtained using the passive stereo method and the surface normal information obtained using the fringe interval are the relationship between differential and integral. That is, the integral of the surface normal is generally the same as the distance information, or the surface normal is generally the same as the differential of the distance information. Therefore, the distance / shape calculation unit 153 compares the distance information and the surface normal, and the deviation can be reduced by eliminating obvious error values ​​and averaging. Alternatively, the information obtained by integrating the surface normal is used for the planar surface portion, and the distance information is used at the level difference portion to obtain the level difference amount, and the deviation can be suppressed accordingly. In this way, the primary distance information and the surface normal information are integrated, thereby obtaining high-precision distance and shape information relative to the distance and shape information obtained using a single measurement method. Note that the order of step S1512 and step S1513 can be interchanged.

[0069] In addition, here, Fig. 7A As shown, a structure may also be such that the processing in step S1516 is additionally performed so that the pattern used when obtaining the primary distance information using the passive stereo method is changed relative to the pattern used when obtaining the surface normal information based on the fringe interval. Note that Fig. 7AEach process in the flow is also executed by the control unit 151 of the measuring device 150 or by a block that operates according to the control of the control unit 151. In step S1516, the control unit 151 instructs the projection device 110 to project a pattern via the communication unit, and also instructs the camera system 120 to capture an image in a state where the pattern is projected via the communication unit, and reacquires the parallax image pair obtained by capturing from the camera system 120. Here, it is desirable that the stripe interval of the pattern used when acquiring the surface normal information in step S1516 is smaller than the stripe interval of the pattern used when obtaining the primary distance information in step S1511. Accordingly, spatially dense surface normal information can be acquired. When acquiring the surface normal information, orthogonal stripe patterns can be switched and projected, or a double cross pattern or a dotted line pattern obtained by combining orthogonal stripe patterns can be projected. According to such a structure, normals in the axial directions orthogonal to each other can be obtained, whereby surface normal information related to the object can be obtained with higher accuracy.

[0070] In addition, a pattern can be selected when acquiring surface normal information based on the primary distance information. For example, as the distance from the projection device increases, the projected pattern image becomes coarse, and as the distance decreases, the projected pattern image becomes fine, thereby controlling the pattern according to the distance so as to achieve the stripe interval required for acquiring surface normal information. In addition, the surface inclination direction can be roughly grasped based on the primary distance information, thereby, by selecting and projecting a stripe pattern in which stripes extend in a direction perpendicular to the inclination direction, or by projecting a fine pattern, the surface inclination amount (the inclination amount of the surface normal) can be acquired with high precision.

[0071] In addition, the pattern used in the passive stereo method can also be selected based on the surface normal information. When the surface normal is tilted, the stripe pattern image becomes too rough, the parallax images cannot be compared with each other, and the distance measurement accuracy is reduced, thereby increasing the fineness of the projected pattern for the tilted surface. Accordingly, high-precision distance information can be obtained. Therefore, a pattern suitable for the shape of the subject can be projected, thereby obtaining distance information with higher accuracy.

[0072] Furthermore, it is desirable that the pattern when acquiring the surface normal information is a stripe pattern in which line segments extending in a direction perpendicular to the straight line connecting the viewpoints of the projection device 110 and the camera system 120 are arranged approximately periodically in the direction of the straight line. According to such a structure, the fringe interval increases with respect to the change in the surface normal, whereby the surface normal can be detected with high accuracy.

[0073] In addition, if Figure 7B As shown, distance information obtained using pattern projection stereo can also be used. Note that Figure 7BEach process in the flow is also executed by the control unit 151 of the measuring device 150 or by a block operating according to the control of the control unit 151.

[0074] In step S1517 , the relative position calculation unit 154 calculates the position and orientation of the projection device 110 relative to the imaging system 120 based on the primary distance information about the object and the position of the pattern image on the parallax image.

[0075] Next, in step S1518 , the distance / shape calculation unit 153 calculates secondary distance information about the object using pattern projection stereo based on the calculated position and posture of the projection device 110 , the position and posture of the camera system 120 , and the position of the pattern image on the parallax image.

[0076] In step S1514, the distance / shape calculation unit 153 compares the primary distance information, the secondary distance information, and the surface normal information, and performs elimination and averaging of error values, thereby obtaining more stable and more accurate distance and shape information. As described above, in the present embodiment, one distance and shape information is calculated by integrating the primary distance information obtained using the passive stereo method and the surface normal information obtained by using the fringe interval, and as a result, more accurate distance and shape information can be generated.

[0077] [Fourth embodiment]

[0078] In this embodiment, the boundary area of ​​the object is extracted by analyzing the continuity of the stripes in the image based on the stripe pattern projected by the projection device 110, and this information is used to obtain the distance and shape information with high accuracy. Fig. 8A Describe the processing flow according to this embodiment. Note that Fig. 8A Each process in the flow is also executed by the control unit 151 of the measuring device 150 or by a block operating according to the control of the control unit 151.

[0079] In step S1521 , the control unit 151 instructs the projection device 110 to project a fringe pattern via the communication unit, instructs the camera system 120 to capture an image with the fringe pattern projected via the communication unit, and acquires a parallax image pair obtained by the capture from the camera system 120 .

[0080] Next, in step S1522, the distance / shape calculation unit 153 generates boundary portion information of the object. Figure 6BIn the figure, the stripes in the region 210 which is a planar surface are lines of the same direction, and in the stripes in the region 211 which is a curved surface, the direction of the line changes but the change is small. On the other hand, in the region (boundary portion) 212 where the distance changes more, the direction of the line changes sharply. Figure 8B An example of an image of a subject consisting of a planar surface and a planar surface object located near and separated from the planar surface in the depth direction is shown, and in a boundary portion 222 of the subject including a horizontal difference in the depth direction, the stripe line is disconnected and discontinuous. In this way, the stripe continuity is analyzed, and thus an area with a large distance change (i.e., the boundary portion of the subject) can be extracted.

[0081] In step S1523 , the distance / shape calculation unit 153 calculates primary distance information of the object from the captured parallax images using the passive stereo method.

[0082] Next, in step S1524, the distance / shape calculation unit 153 generates high-precision distance and shape information from the primary distance information using the boundary information of the subject. In a boundary area where the distance changes greatly, the accuracy of the stereo method is reduced in principle. Therefore, for the extracted boundary area, without using the primary distance information obtained by the stereo method, the distance and shape information is generated by extending the primary distance information of each area in the area sandwiching the boundary area to the boundary part. For example, Figure 8B In the example shown, with respect to the boundary portion 222, the primary distance information related to the region 221 is used on the region 221 side, and the primary distance information related to the region 220 is used on the region 220 side. Figure 6B In the example shown, primary distance information related to region 210 is used for region 210 side with respect to boundary portion 212, and primary distance information is generated for region 211 side so that the surface normal in region 211 changes continuously. In this way, the boundary region of the subject is extracted based on the stripe shape and distance information is obtained based on this result, whereby distance and shape information can be obtained with high accuracy even for the boundary portion where the distance change is large. Here, secondary distance information obtained by the pattern projection stereo method and surface normal information obtained based on the stripe interval are used, whereby distance and shape information related to the subject can be obtained with higher accuracy.

[0083] Then, in step S1525, the control unit 151 stores the calculated high-precision distance and shape information in a storage unit (not shown), or outputs it to an external device via a recording unit or a communication unit or the like.

[0084] As described above, according to the present embodiment, stripe pattern light is projected by the projection device 110, and the boundary area of ​​the subject is extracted by analyzing the continuity of the stripes in the image including the subject and the pattern light. The distance and shape information can be obtained with higher accuracy by using information related to the boundary area.

[0085] [Other embodiments]

[0086] The present invention can be implemented by supplying a program for processing the functions according to the above-described typical embodiments to a system or device via a network or storage medium and causing one or more processors in a computer of the system or device to read and execute the processing of the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0087] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, the following claims are attached to inform the public of the scope of the present invention.

[0088] This application claims priority to and the benefit of Japanese Patent Application No. 2019-122099, filed on Jun. 28, 2019, and Japanese Patent Application No. 2020-106407, filed on Jun. 19, 2020, which are hereby incorporated by reference herein in their entirety.

Claims

1. A measuring device comprising: a projection device configured to project a predetermined pattern on a subject; as well as a camera system configured to capture a group of images at two viewpoints arranged at different positions, wherein each of the two viewpoints is arranged at a position where the distance from the other viewpoint is shorter than the distance from the projection device, The measuring device also includes: a relative position calculation unit, configured to obtain a relative position of the projection device relative to at least one of the two viewpoints based on a position of the pattern image on the image group and a positional relationship between the two viewpoints; and The distance obtaining unit is configured to obtain the distance information of the object based on the relative position and the pattern image position on the image at the viewpoint where the relative position has been obtained.

2. The measuring device according to claim 1, wherein: The position and posture of the projection device are changeable.

3. The measuring device according to claim 1, wherein: The projection device is arranged at a position on an extension line of a straight line connecting the two viewpoints.

4. The measuring device according to claim 1, further comprising: A control unit, used to control the operation of the projection device, The control unit makes the pattern to be projected by the projection device different when the relative position is obtained and when the distance information is obtained.

5. The measuring device according to claim 4, wherein: The control section causes the projection device to project a stripe pattern as the pattern, in which line segments extending in a direction perpendicular to a straight line connecting the two viewpoints are arranged substantially periodically in the direction of the straight line.

6. The measuring device according to claim 5, wherein: The color and / or shape of at least one line segment in the stripe pattern is different from the colors and / or shapes of the other line segments.

7. The measuring device according to claim 5, wherein: The distance obtaining section obtains surface normal information at a local position of the object based on a local fringe interval of a pattern image of the fringe pattern at the local position of the image.

8. The measuring device according to claim 7, wherein: The distance obtaining section obtains the distance information of the object based on the surface normal information, the relative position, and the pattern image position on the image at the viewpoint at which the relative position has been obtained.

9. The measuring device according to claim 7, wherein: The distance obtaining section obtains the distance information of the object based on the surface normal information, the relative position, the pattern image position on the image at the viewpoint at which the relative position has been obtained, and the distance between the two viewpoints.

10. The measuring device according to claim 7, wherein: The control section causes the projection device to project a pattern having a smaller fringe interval when obtaining the surface normal information than when obtaining the relative position.

11. The measuring device according to any one of claims 1 to 10, wherein: The pattern is a pattern including line segments, and The distance obtaining section extracts a boundary region of the subject based on the shape of the pattern image of the line segment on the image at the viewpoint whose relative position has been obtained, and obtains the distance information of the subject based on information related to the boundary region.

12. The measuring device according to claim 1, wherein: The imaging system divides light incident through a single optical system to capture a group of images at the two viewpoints.

13. A camera device, comprising: The measuring device according to any one of claims 1 to 12; as well as The camera system further acquires images taken without projecting the pattern.

14. A measurement system comprising: a projection device configured to project a predetermined pattern on a subject; a camera system configured to capture a group of images at two viewpoints arranged at different positions; as well as a measuring device connected to the projection device and the camera system, wherein each of the two viewpoints is arranged at a position where the distance from the other viewpoint is shorter than the distance from the projection device, The measuring device also includes: a relative position calculation unit, configured to obtain a relative position of the projection device relative to at least one of the two viewpoints based on a pattern image position on the image group and a positional relationship between the two viewpoints; and A distance obtaining section is configured to obtain distance information of the object based on the relative position and the pattern image position on the image at the viewpoint where the relative position has been obtained.

15. A method for controlling a measuring device, the measuring device comprising a projection device and a camera system, the projection device being configured to project a predetermined pattern on a subject, the camera system being configured to capture an image group at two viewpoints arranged at different positions, each of the two viewpoints being arranged at a position where the distance from the other viewpoint is shorter than the distance from the projection device, The control method comprises: a relative position calculation step for obtaining the relative position of the projection device relative to at least one of the two viewpoints based on the pattern image position on the image group and the positional relationship between the two viewpoints; as well as A distance obtaining step is used to obtain distance information of the object based on the relative position and the pattern image position on the image at the viewpoint where the relative position has been obtained. 16 . A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a computer, the computer program implements the steps of the control method of the measuring device according to claim 15 .

Images