Three-dimensional measuring device

Abstract

This three-dimensional measuring device comprises a projecting unit that projects a prescribed striped pattern toward a target region, an imaging unit that images an object, on which the prescribed striped pattern is projected, and outputs event data, a measuring unit that measures the three-dimensional shape of the object, and a projection control unit that controls the projection unit, wherein: the measuring unit comprises a determining unit that determines whether a captured image generated from the event data includes a region in which specular reflection is occurring, a detecting unit that detects an imaging pixel group, of the imaging unit, that imaged the region in which the specular reflection is occurring, in the captured image that is determined, by the determining unit, to include a region in which specular reflection is occurring, and an identifying unit that identifies a projection pixel group, of the projecting unit, that is a projection region corresponding to an imaging region formed by the imaging pixel group detected by the detecting unit; and if the determining unit has determined that a region in which specular reflection is occurring is included, the projection control unit projects the prescribed striped pattern with the projection pixel group identified by the identifying unit in an extinguished state.

Description

Three-dimensional measuring device

[0001] The present disclosure relates to a three-dimensional measuring apparatus that measures the three-dimensional shape of a measurement object.

[0002] Conventionally, as a three-dimensional measurement device for measuring the three-dimensional shape of a measurement object, for example, a device using a phase shift method is known. The phase shift method is a technique for projecting a plurality of phase-shifted stripe pattern images onto a measurement object onto which the plurality of stripe pattern images are projected, thereby performing three-dimensional measurement.

[0003] Regarding technology for performing three-dimensional measurement using the phase shift method, a three-dimensional measurement device is known, as disclosed in Patent Document JP2021067644A, for generating images of a measurement object more quickly. This three-dimensional measurement device employs a sine wave pattern as the predetermined stripe pattern for the phase shift method. Furthermore, an event camera is employed that outputs event data including two-dimensional point data that identifies the positions of pixels that experience a change in brightness when receiving light. From the event data, a captured image of the measurement object onto which the stripe pattern is projected is generated. The event camera is characterized by not outputting pixel information without brightness changes, i.e., redundant data, as output by conventional cameras. This reduces data traffic and image processing, enabling faster acquisition of information about the shape of the measurement object. Meanwhile, the event data does not include the brightness information used in the phase shift method. Therefore, brightness information (i.e., stripe pattern information) is calculated based on the time difference between the occurrence of event data with a positive brightness change (i.e., positive polarity event data) output on a pixel-by-pixel basis when light is projected and the occurrence of event data with a negative brightness change (i.e., negative polarity event data) output when the light is turned off. As a result, the three-dimensional shape of the measurement object is measured using the event data.

[0004] Incidentally, measurement targets that are prone to specular reflection, such as aluminum plates, may be captured using an event camera for 3D measurement. In 3D measurement using an event camera, the 3D measurement is performed by utilizing the brightness changes at each image pixel when a measurement stripe pattern is projected onto the object. Therefore, if specular reflection occurs during imaging, event data that becomes noise over a wide area of ​​the captured image is likely to be generated, potentially reducing measurement accuracy. While it is possible to physically hide the area where specular reflection occurs, it is not practical to cover that area for each measurement target.

[0005] The present disclosure has been made to solve the above-mentioned problems, and its purpose is to provide a configuration that can improve the measurement accuracy of a measurement object that is prone to specular reflection.

[0006] a projection control unit that controls the projection unit; a determination unit that determines whether an image generated from the event data includes an area where specular reflection occurs; a detection unit that detects an image group of the image capturing unit that captured the area where specular reflection occurs in the image captured by the determination unit; and a projection control unit that determines that the image capturing area formed by the image capturing pixel group detected by the detection unit is a projection area; and when the determination unit determines that the area where specular reflection occurs is included, the projection control unit projects the predetermined stripe pattern in which the image capturing area includes the area where specular reflection occurs.

[0007] As a result, when specular reflection occurs, the projection pixel group whose projection area is the imaging area of ​​the imaging pixel group that captured the area where the specular reflection occurs is turned off. Therefore, although the imaging pixel group whose imaging area is the projection area of ​​the identified projection pixel group does not output event data used for measurement, noise caused by specular reflection in the imaging pixels surrounding the imaging pixel group is reduced. This makes it possible to improve the measurement accuracy of measurement objects that are prone to specular reflection.

[0008] 1 is a block diagram illustrating a schematic configuration of a three-dimensional measurement apparatus according to a first embodiment; FIG. 1 is a diagram illustrating a state in which a stripe pattern for a general phase shift method is projected onto a measurement object; FIG. 2 is a diagram illustrating three-dimensional measurement using the phase shift method; FIG. 3 is a diagram illustrating the relationship between the brightness value and the time difference between the output timing of positive polarity event data and negative polarity event data output per unit time; FIG. 4 is a diagram illustrating a state in which a projection aperture is imaging a measurement object undergoing specular reflection; FIG. 5 is a diagram illustrating the output timing of negative polarity event data when no specular reflection occurs; FIG. 6 is a diagram illustrating the output timing of negative polarity event data when specular reflection occurs; FIG. 7 is a diagram illustrating an example of a first stripe pattern generated so that non-imaging regions are the darkest; FIG. 8 is a diagram illustrating an example of a portion of a second stripe pattern in which a specified projection pixel group is in an off state; FIG. 9 is a flowchart illustrating the flow of three-dimensional measurement processing according to the first embodiment; FIG. 10 is a diagram illustrating an imaging result when the second stripe pattern is not projected; FIG. 11 is a diagram illustrating an imaging result when the second stripe pattern is projected; FIG. 12 is a diagram illustrating the sum of the event amounts of imaging pixel groups having the same y coordinate; and FIG. 13 is a diagram illustrating the sum of the event amounts of imaging pixel groups having the same x coordinate. 14A is a diagram illustrating a predetermined region to be detected as an imaging pixel group that captures an area where specular reflection occurs in the second embodiment. FIG. 14B is a diagram illustrating epipolar lines on a projection plane obtained by converting imaging pixel coordinates on the imaging plane. FIG. 14C is a diagram illustrating a portion of a projection image to which line segments of the epipolar lines have been added. FIG. 14D is a diagram illustrating a projection pixel group identified by a projection region obtained by thickening the epipolar line in FIG. 14A. FIG. 14E is a diagram illustrating a portion of a projection image to which line segments of a plurality of epipolar lines have been added in a modified example of the third embodiment. FIG. 15A is a diagram illustrating a projection pixel group including the line segments of each epipolar line in FIG. 15A. FIG. 15F is a diagram illustrating the hardware configuration of a three-dimensional measurement device according to an embodiment of the present disclosure.

[0009] First Embodiment A three-dimensional measuring apparatus according to a first embodiment of the present disclosure will be described below with reference to the drawings.

[0010] The three-dimensional measuring device 10 according to this embodiment is a device that measures the three-dimensional shape of a measurement object R1. As shown in FIGS. 1 and 2 , the three-dimensional measuring device 10 includes a control unit 11, a projection unit 20, an imaging unit 30, and a measurement unit 40. The control unit 11 is responsible for overall control. The projection unit 20 projects a predetermined stripe pattern for three-dimensional measurement onto the measurement object R1. The imaging unit 30 captures an image of the measurement object R1 onto which the predetermined stripe pattern is projected. The measurement unit 40 measures the three-dimensional shape of the measurement object R1 from the captured image. The three-dimensional measuring device 10 configured in this manner measures the three-dimensional shape of the measurement object R1, such as a workpiece, that is attached to a robot hand and moves relatively to the hand at high speed. Here, relative movement refers to the relative movement between the movement of the three-dimensional measuring device 10 attached to the robot hand and the movement of the measurement object R1. When the position of the three-dimensional measuring apparatus 10 is fixed, the relative movement is the movement of the measurement object R1.

[0011] For convenience, Fig. 2 shows a simplified version of a typical stripe pattern with up to 13 stripes. More specifically, typical stripe patterns are represented as sine wave patterns, so the light and dark portions of the stripe pattern have the same width. However, for convenience, Fig. 2 shows the dark portions with a smaller width as lines. Furthermore, although the number of stripes is 13 or more in the embodiment, it is abbreviated to 13.

[0012] As shown in FIG. 16 , the three-dimensional measuring apparatus 10 includes a processor 201 and a memory 203 as a hardware configuration. For example, the three-dimensional measuring apparatus 10 may include a microcomputer. The microcomputer may include a CPU (Central Processing Unit), a system bus, an input / output interface, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile memory, etc. The memory 203 pre-stores, in addition to a program related to robot control, a program related to control of the projection unit 20, and a program for executing control processing using the three-dimensional measurement results obtained by the measurement unit 40. The functions of the control unit 11 and the measurement unit 40 may be realized by the above hardware configuration.

[0013] The projection unit 20 is a so-called DLP (registered trademark) projector. The projection unit 20 is controlled by the control unit 11 to project a predetermined stripe pattern by reflecting light from a light source using a DMD (Digital Micromirror Device) element. The DMD element includes minute mirrors corresponding to each pixel (i.e., projection pixel) of an image projected onto a screen, and the minute mirrors are arranged in an array. The DMD element changes the angle of each mirror to switch (i.e., turn ON / OFF) the light emitted to the screen in microsecond increments. By switching each mirror from reflection OFF to reflection ON, the projection pixel is switched to a light-emitting state. Furthermore, by switching each mirror from reflection ON to reflection OFF, the projection pixel is switched to an unlit state. That is, the projection unit 20 projects a predetermined stripe pattern by controlling, by the control unit 11, the ON / OFF reflection of incident light by the DMD, which is an array of multiple mirrors. Therefore, the control unit 11 changes the gradation (i.e., brightness) of the reflected light depending on the ratio of the time each mirror is turned on to the time it is turned off. This enables gradation display based on the image data of the image to be projected. In this embodiment, the projection unit 20 has mirrors corresponding to k×l projection pixels (e.g., 1140×912), with the upper left being (1,1) and the lower right being (k,l).

[0014] In this configuration, the longer the emission time (i.e., the time from reflection ON to reflection OFF) of a single pulse emitted once within a unit time secured for each emission state, the brighter the emission state becomes. Therefore, the emission state can be identified based on the emission time. For example, consider a case where R (red), G (green), and B (blue) colors are provided as light incident on the DMD element. In this case, an R-color emission state in which R is emitted by reflecting off a mirror, a G-color emission state in which G is emitted by reflecting off a mirror, and a B-color emission state in which B is emitted by reflecting off a mirror are repeated at a predetermined short interval. By individually adjusting the emission time of each emission state, a color image can be projected. Therefore, the control unit 11 sets the reflection ON / OFF timing within the unit time for each mirror according to a predetermined stripe pattern described below. Note that the control unit 11, which controls the projection unit 20, can be an example of a projection control unit.

[0015] The imaging unit 30 is a so-called event camera. The imaging unit 30 includes an imaging element that outputs event data (specifically, data including two-dimensional point data, time, and the polarity of the brightness change) including two-dimensional point data that identifies the position of an imaging pixel that experienced a brightness change upon receiving light. The imaging unit 30 generates an image from the event data output from the imaging element. Therefore, for each captured image, the imaging unit 30 outputs positive-polarity (i.e., positive brightness change) event data when a brightness change resulting in a brighter image occurs due to the reception of light, and outputs negative-polarity (i.e., negative brightness change) event data when a brightness change resulting in a darker image occurs due to the disappearance of the light. Image data of the measurement object R1 is generated by plotting the two-dimensional point data of multiple event data output within a certain period of time as points on a predetermined plane. The imaging unit 30 outputs the generated image data or event data to the measurement unit 40.

[0016] The measurement unit 40 is controlled by the control unit 11. The measurement unit 40 performs three-dimensional measurement processing to measure the three-dimensional shape of the measurement object R1 based on an image captured by the imaging unit 30 of the measurement object R1 onto which a predetermined stripe pattern is projected from the projection unit 20. The three-dimensional measurement processing in this embodiment employs a phase shift method.

[0017] Generally, a phase value θ corresponding to a distorted value according to the surface shape of the measurement object R1 is determined based on a grating image (i.e., a fringe image) obtained by capturing an image of the measurement object R1 onto which a fringe pattern for the phase shift method is projected. A sine wave pattern specified by the brightness value I(x, y, n) in the following equation (1) is used. In this fringe pattern, the brightness periodically changes in a first direction and remains constant in a second direction perpendicular to the first direction. When the number of phase shifts is N, the brightness values ​​I(x, y, n) of N phase-shifted grating images (i.e., fringe images) are expressed by equation (1): I(x, y, n) = a(x, y) cos{θ(x, y) + 2πn / N} + b(x, y) (1) Here, point (x, y) is one point (i.e., one pixel) in the grating image. a(x, y) is the brightness amplitude. b(x, y) indicates the background brightness. θ(x, y) indicates the phase value of the grating where n = 0. The distance z to the point (x, y) is measured according to the phase value θ(x, y) calculated from the brightness values ​​I(x, y, n) of the N grating images.

[0018] Specifically, consider a case where three grating images are obtained in one cycle formed by the above-described R, G, and B light emission states. In this case, N=3, and the luminance value I(x, y, 0) in the R light emission state, the luminance value I(x, y, 1) in the G light emission state, and the luminance value I(x, y, 2) in the B light emission state are obtained from the captured image. In this case, the stripe pattern projected from the projection unit 20 is configured so that the phases of the sine wave pattern consisting of only R, the sine wave pattern consisting of only G, and the sine wave pattern consisting of only B are shifted by 2π / 3.

[0019] By obtaining the luminance values ​​I(x, y, 0), I(x, y, 1), and I(x, y, 2) at the point (x, y) in the captured image, the phase value θ(x, y) can be calculated using the above formula (1). The distance z to the point (x, y) is measured according to the phase value θ(x, y) calculated in this manner. By measuring the distance z to each point (x, y) of the captured measurement object R1 in this manner, the three-dimensional shape of the measurement object R1 can be measured.

[0020] For example, when determining the distance z from the three-dimensional measurement device 10 to point Pr1 in FIG. 3 , the measurement unit 40 determines the phase value θ of point Pr1 and information on which stripe pattern point Pr1 is located (i.e., the stripe number) from N captured images captured by the image capture unit 30 after the projection unit 20 shifts and projects a stripe pattern N times. From the phase value θ and stripe number thus determined, the angle θp1 at the projection unit 20 and the angle θc1 at the image capture unit 30 are determined. Since the distance between the projection unit 20 and the image capture unit 30 (i.e., the parallax Lpc) is known, the distance z to point Pr1 can be determined by triangulation. Similarly, the distance z to point Pr2 in FIG. 3 can be determined by triangulation based on the angle θp2 at the projection unit 20 and the angle θc2 at the image capture unit 30, which are determined from the phase value θ of point Pr2 determined from the N captured images and the stripe number. Three-dimensional measurement can be performed by performing this calculation over the entire measurement area.

[0021] In this embodiment, an event camera is used as an imaging unit to accurately capture the measurement target R1, which is moving relatively at high speed. With this configuration, event data corresponding to pixels where a luminance change occurs is output. However, since the event data does not include luminance values, it is not possible to directly obtain the luminance values ​​required for the phase shift method (e.g., I(x, y, 0), I(x, y, 1), and I(x, y, 2) described above).

[0022] For this reason, in this embodiment, stripe pattern information Is(x,y,n) corresponding to luminance value I(x,y,n) is calculated as the time difference between output timing t1 of the first positive polarity event data (see the upward arrow in FIG. 4 ) and output timing t2 of the next negative polarity event data (see the downward arrow in FIG. 4 ) in unit time T, as shown in FIG. 4 . After positive polarity event data is output at emission start timing t1, negative polarity event data is output at emission end timing t2. In other words, the longer the time difference, the higher the luminance value at that imaging pixel.

[0023] The stripe pattern information Is(x, y, n) obtained in this manner is affected by the three-dimensional shape of the measurement object R1. Therefore, the phase value θ(x, y) can be obtained using the following equation (2): Is(x, y, n) = a(x, y) cos {θ(x, y) + 2πn / N} + b(x, y) (2)

[0024] Next, the three-dimensional measurement process performed by the measurement unit 40 and the stripe pattern projected from the projection unit 20 in this embodiment will be described in detail.

[0025] When performing three-dimensional measurement of a measurement object that is prone to specular reflection, such as an aluminum plate, if specular reflection occurs during imaging, event data that becomes noise over a wide area of ​​the captured image is likely to occur. For example, as illustrated in Figure 5, when performing three-dimensional measurement of a measurement object R1 that includes an aluminum plate, the measurement object R1 is imaged in a state where a mirror image formed by the mirror surface of the measurement object R1 and a real image overlap. If the projection opening 21 of the projection unit 20 is reflected in this mirror image, the image of the projection opening 21 may cause the output of event data that becomes noise for the following reasons.

[0026] As described above, the stripe pattern for the phase shift method is a pattern in which the luminance changes periodically in a first direction and does not change in a second direction perpendicular to the first direction. The projection unit 20 that projects such a stripe pattern is controlled so that the number of projected pixels that change from a lit state to an extinguished state for each stripe within a unit time increases stepwise. Each imaging pixel of the imaging unit 30 that captures the projected stripe pattern outputs event data for both the lit and extinguished states within the unit time (see FIG. 4).

[0027] Then, in the imaging pixel group capturing the projection aperture 21 (i.e., the mirror image) of the projection unit 20 that projects the stripe pattern, a luminance change occurs in response to the luminance change in the stripe pattern. Specifically, the luminance change in the stripe pattern occurs as the projection pixels change from an on state to an off state within the unit time. At the same time, as the projection pixels change from an on state to an off state within the unit time, a portion of the light emitted from the projection aperture 21 disappears. Therefore, a negative luminance change occurs in the imaging pixel group capturing the area of ​​the projection aperture 21 from which the portion of light was emitted. This negative luminance change occurs for each area where the light disappears in response to the luminance change in the stripe pattern. As a result, the area in the captured image that captures the projection aperture 21 flashes multiple times within the unit time, which can cause the image of the specularly reflected projection aperture 21 to output noisy event data. Furthermore, the luminance of the imaging element that receives the specularly reflected light from the projection aperture 21 (i.e., the mirror image) is easily saturated. Furthermore, because the projection aperture 21 is not in focus, the projection aperture 21 is captured as blurred and wide. These may also cause noise event data to be output.

[0028] For this reason, in this embodiment, a stripe pattern (hereinafter also referred to as the first stripe pattern P1) is used to detect the group of imaging pixels of the imaging unit 30 that captured the area in the captured image where specular reflection occurs, and a stripe pattern (hereinafter also referred to as the second stripe pattern P2) is used to eliminate the influence of the detected specular reflection.

[0029] First, the first stripe pattern P1 will be described with reference to FIGS.

[0030] A pattern that gradually becomes brighter from left to right is projected onto a non-specularly reflective plane from the projection unit 20. At each horizontal pixel on a certain horizontal line of the imaging unit 30, negative polarity event data is output at the timing for each group of horizontal pixels indicated by the solid lines in Figure 6A.

[0031] On the other hand, consider a case where a blurred, wide image of the specularly reflected projection aperture 21 is captured when the same pattern as in Figure 6A is projected. In this case, as shown in Figure 6B, the group of imaging pixels of the imaging unit 30 that captures the specularly reflected projection aperture 21 (e.g., horizontal pixels x1-x2 in Figure 6B) outputs negative event data at the same timing as the other horizontal pixels. This is because the light emitted by the projection aperture 21 changes to cause a change in brightness in the projected stripe pattern. In the imaging pixel group capturing the blurred, wide projection aperture 21, a change in brightness occurs regardless of the location of the projected pattern (i.e., any horizontal pixel). Event data is output due to the change in brightness in the imaging pixel group capturing the projection aperture 21.

[0032] Also, consider a case where a luminance change occurs in a group of projection pixels of the projection unit 20, whose projection area is a predetermined area (hereinafter also referred to as a non-imaging area) outside the imaging area of ​​the imaging unit 30. Even in this case, the imaging pixel group that images the projection opening 21 that is specularly reflected as described above can output event data due to the luminance change in the projection pixel group.

[0033] In this embodiment, the first stripe pattern P1 is generated so that the determination timing, which is the timing at which the non-image capture area is turned off, is earlier than the timing at which all projected pixels in the image capture area of ​​the image capture unit 30 are turned off. That is, the region of the first stripe pattern P1 that is the image capture area of ​​the image capture unit 30 is a stripe pattern for a conventional phase shift method. In a stripe pattern for a conventional phase shift method, the non-image capture area outside the image capture area of ​​the image capture unit 30 is the darkest (i.e., turns off earliest). For example, as shown in Scene 01 of FIG. 7 , the image capture area E2 of the image capture unit 30 is included within the projection area E1 of the first stripe pattern P1. The non-image capture area E3 is located around the image capture area E2. In this case, the first stripe pattern P1 first darkens the non-image capture area E3, as shown in Scene 02 of FIG. 7 . Then, as shown in Scenes 03 to 05 of FIG. 7 , the first stripe pattern P1 is projected so that the dark area in the image capture area E2 gradually increases.

[0034] This allows the imaging pixel group that outputs negative event data in response to the determination timing to be detected as the imaging pixel group that captured an area where specular reflection occurs (hereinafter also referred to as the first imaging pixel group). In particular, at the determination timing, none of the projection pixels projected into the imaging area are turned off. That is, most of the area captured by the imaging pixels is bright. Therefore, the detection accuracy of the first imaging pixel group can be improved compared to when most of the area captured by the imaging pixels is dark. This is because the sensitivity of the imaging pixels that capture dark areas is higher than the sensitivity of the imaging pixels that capture bright areas. That is, the imaging pixels that capture dark areas are more likely to produce noise than the imaging pixels that capture bright areas.

[0035] Next, the second stripe pattern P2 will be described with reference to FIG.

[0036] In the second stripe pattern P2 in this embodiment, a specific projection pixel group is in an off state. The projection pixel group of the projection unit 20, whose projection area is the imaging area of ​​the first imaging pixel group detected as described above as the imaging pixel group that captured the area where specular reflection occurs, is identified as the specific projection pixel group.

[0037] Specifically, for example, in the second stripe pattern P2, as shown in Fig. 8 , in each of the N stripe patterns for the conventional phase shift method, the projection region of the projection pixel group identified as described above (e.g., the area Po indicated by the two-dot chain line in Fig. 8 ) is in an off state. As a result, the first imaging pixel group, whose imaging region is the projection region of the identified projection pixel group, does not output event data used for measurement. However, it is possible to reduce noise that may occur in the imaging pixels surrounding the first imaging pixel group due to specular reflection.

[0038] The processing performed by the three-dimensional measuring apparatus 10 in this embodiment will be described in detail below with reference to the flowchart shown in FIG.

[0039] When the three-dimensional measurement process is started in response to a predetermined operation, a process for projecting the first stripe pattern is performed as shown in step S101. In this process, the projection unit 20 controlled by the control unit 11 projects the first stripe pattern P1 generated as described above through the projection aperture 21.

[0040] Next, a process for acquiring event data is performed as shown in step S102. While the first stripe pattern P1 is being projected as described above, positive and negative event data output from each image sensor of the image capturing unit 30 within a unit time is acquired.

[0041] Next, in the determination process of step S103, it is determined whether specular reflection is occurring. Specifically, if it is determined that negative polarity event data is not being output at the determination timing, it is determined that specular reflection is not occurring (No in step S103). In this case, a process for outputting data shown in step S108 is performed. In the process of step S108, the event data acquired as described above is output to the higher-level device. Thereafter, the three-dimensional measurement process for the unit time ends. Note that the measurement unit 40 that performs the determination process of step S103 above may be an example of a determination unit.

[0042] On the other hand, if it is determined that negative event data is being output at the determination timing, it is determined that specular reflection is occurring (Yes in step S103). In this case, the process of detecting the first imaging pixel group shown in step S104 is performed. In this process, as described above, the imaging pixel group for which negative event data is being output at the determination timing in the first stripe pattern P1 is detected as the first imaging pixel group. Note that the measurement unit 40 that performs the process of step S104 may be an example of a detection unit.

[0043] Next, a process for identifying a projected pixel group is performed in step S105. A projected pixel group of the projection unit 20, whose projection region is the imaging region of the first imaging pixel group detected as described above, is identified. For example, as will be described later, the projected pixel group can be identified by performing a geometric calculation using an epipolar line defined from the positional relationship between the imaging unit 30 and the projection unit 20. Note that the measurement unit 40 that performs the process of step S105 above may be an example of an identification unit.

[0044] Next, a process for projecting the second stripe pattern is performed in step S106. In this process, the projection unit 20, controlled by the control unit 11, projects the second stripe pattern P2, in which the projection pixel group identified as described above is in an off state, through the projection aperture 21.

[0045] Then, a process for acquiring event data shown in step S107 is performed. With the second stripe pattern P2 being projected as described above, positive and negative polarity event data output from each imaging element of the imaging unit 30 within a unit time is acquired. In this process, event data from the first imaging pixel group is not acquired, but event data that is not affected by specular reflection can be acquired from other imaging pixel groups. Then, a process for outputting data shown in step S108 is performed. The event data acquired as described above is output to the higher-level device. Thereafter, the three-dimensional measurement process for the unit time ends.

[0046] For example, consider a case where three-dimensional measurement of a transparent object is performed using an event camera. When the second stripe pattern P2 is not projected as in the conventional method due to the influence of specular reflection, a wide range of defects occurs in the captured image, as illustrated in FIG. 10A. In contrast, when the second stripe pattern P2 is projected as described above, defects are less likely to occur in the captured image, as illustrated in FIG. 10B. The area So indicated by the two-dot chain line in FIG. 10B is the area of ​​the first imaging pixel group from which no event data is output.

[0047] As described above, in the three-dimensional measuring apparatus 10 according to this embodiment, a first imaging pixel group is detected in a captured image in which specular reflection is determined to occur (step S104). A projection pixel group of the projection unit 20, whose projection area is the imaging region of the detected first imaging pixel group, is identified (step S105). If it is determined that specular reflection is occurring (Yes in step S103), the projection unit 20 is controlled to project the second stripe pattern P2 in which the identified projection pixel group is in an off state (step S106).

[0048] As a result, when specular reflection occurs, the second stripe pattern P2 is projected in a state where the projection pixel group whose imaging area is the imaging area of ​​the first imaging pixel group that captured the area where specular reflection occurs is in an off state. Therefore, the first imaging pixel group whose imaging area is the projection area of ​​the identified projection pixel group does not output event data used for measurement. However, noise that may occur due to specular reflection in the imaging pixels surrounding the first imaging pixel group is reduced. This improves the measurement accuracy of the measurement object R1, which is prone to specular reflection.

[0049] In particular, the first stripe pattern P1 is generated so that within a unit time, the determination timing, which is the turn-off timing of a projection pixel group whose projection area is a predetermined area outside the imaging area of ​​the imaging unit 30, is earlier than the turn-off timing of any projection pixel whose projection area is the imaging area of ​​the imaging unit 30. Then, the imaging pixel group from which event data is output in accordance with the determination timing is detected as the first imaging pixel group.

[0050] When capturing an image of a stripe pattern that is projected so that the number of projection pixels that change from a lit state to an unlit state for each stripe increases stepwise within a unit time, each imaging pixel of the imaging unit 30 outputs event data for both the lit and unlit states within the unit time. The group of imaging pixels capturing the projection aperture 21 of the projection unit 20 that projects the stripe pattern experiences luminance changes corresponding to the luminance changes in the stripe pattern. As a result, the area of ​​the captured image that captures the projection aperture 21 flashes multiple times within the unit time, potentially causing the output of event data that contains noise from the specularly reflected image of the projection aperture 21.

[0051] In the imaging pixel group capturing the specularly reflected projection aperture 21, a luminance change occurs regardless of where in the stripe pattern the luminance change occurs. Event data is output multiple times within a unit time due to the luminance change in the imaging pixel group. On the other hand, if specular reflection is not occurring, no luminance change occurs within the imaging area at the determination timing, and therefore no event data is output from any imaging pixel. Therefore, the imaging pixel group that outputs event data in accordance with the determination timing can be detected as the first imaging pixel group that captured the area where specular reflection is occurring.

[0052] Second Embodiment Next, a three-dimensional measuring apparatus according to a second embodiment of the present disclosure will be described with reference to the drawings.

[0053] The present embodiment differs from the first embodiment in that the first imaging pixel group is detected based on the amount of event data generated (i.e., the event amount). Components that are substantially the same as those in the first embodiment are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0054] As described above, the imaging pixel group capturing the specularly reflected projection aperture 21 (i.e., the first imaging pixel group) outputs event data multiple times within a unit time. Therefore, the event volume is larger than that of an imaging pixel group capturing an area where specular reflection does not occur. In particular, the center of the first imaging pixel group is likely to have the largest output event volume.

[0055] 11A and 11B are diagrams illustrating the event amount in an image where specular reflection occurs, with Fig. 11A showing the sum of the event amounts for imaging pixel groups with the same y coordinate and Fig. 11B showing the sum of the event amounts for imaging pixel groups with the same x coordinate. As shown in Figs. 11A and 11B, the event amount increases in both the x and y directions in areas where specular reflection occurs.

[0056] Therefore, in the three-dimensional measurement process of this embodiment, if it is determined that specular reflection is occurring as described above (Yes in step S103), the process of detecting the first imaging pixel group is performed in step S104. In this process, the imaging pixel group included in a predetermined region centered on the imaging pixel with the largest event amount is detected as the first imaging pixel group. The predetermined region is geometrically calculated as shown in FIG. 12 as a mirror image size D (pix) calculated using the aperture diameter (i.e., the lens barrel diameter) Dp of the projection aperture 21 projected onto the sensor surface of the imaging unit 30. Note that W (pix) in FIG. 12 indicates the number of horizontal pixels of the imaging unit 30. θg indicates the angle of view (i.e., the camera angle of view) of the imaging unit 30. The smaller the distance z from the imaging unit 30 to the mirror surface, the larger the mirror image size D. Therefore, it is desirable to set the distance z to the minimum distance zmin specified in the specifications. Furthermore, in this embodiment, the imaging pixel group is detected without using the determination timing. Therefore, in the process of projecting the first stripe pattern shown in step S101, a stripe pattern for the conventional phase shift method is projected.

[0057] In this way, the imaging pixel group detection process in this embodiment detects a first imaging pixel group in which the imaging pixel with the largest event amount is located at the center. Therefore, it is possible to detect an imaging pixel group that captured an area where specular reflection occurs based on the event amount. Even when the first imaging pixel group is detected in this manner, the second stripe pattern P2 is projected in which the projection pixel group whose projection area is the imaging area of ​​the first imaging pixel group is in an off state. This reduces noise that may be generated due to specular reflection, thereby improving the measurement accuracy of the measurement object R1, which is prone to specular reflection.

[0058] Third Embodiment Next, a three-dimensional measuring apparatus according to a third embodiment of the present disclosure will be described with reference to the drawings.

[0059] In this embodiment, a projection pixel is identified based on a line segment on an epipolar line for the coordinates of the central imaging pixel of the first imaging pixel group, and the line segment on the epipolar line has the coordinates of the projection pixels corresponding to the minimum measurement distance and the maximum measurement distance at both ends. The same reference numerals are used to designate components that are substantially the same as those in the second embodiment, and descriptions thereof will be omitted.

[0060] In the three-dimensional measurement process of this embodiment, the process for detecting the first imaging pixel group shown in step S104 determines the coordinates (uc, vc) of the central imaging pixel of the first imaging pixel group on a plane corresponding to the sensor surface of the imaging unit 30 (hereinafter also referred to as imaging plane F1). Then, in the process for identifying the projection pixel group shown in step S105, the projection pixel group is identified based on a line segment of an epipolar line for the imaging pixel coordinates (uc, vc) determined as described above. The epipolar line is a straight line that converts the coordinates (uc, vc) of the imaging pixel into coordinates of a projection pixel on a plane corresponding to the sensor surface of the projection unit 20 (hereinafter also referred to as projection plane F2). The line segment on the epipolar line is a line segment whose both ends are the coordinates of the projection pixels corresponding to the minimum measurement distance and the maximum measurement distance, respectively.

[0061] A method for identifying a projected pixel group on the projection plane F2 corresponding to the coordinates (uc, vc) of an image pixel on the imaging plane F1 in the process for identifying the projected pixel group will be described in detail below with reference to FIG. 13 and other figures.

[0062] When the coordinates (uc, vc) of an image pixel are mapped to spatial coordinates (Xc, Yc, Zc), Xc and Yc at any Zc can be calculated by the following equation (3).

[0063] Then, when the internal matrix Kc of the imaging unit 30 is expressed by the following equation (4), the internal matrix Kp of the projection unit 20 is expressed by the following equation (5), and the rotation matrix R and translation matrix t between the projection unit 20 and the imaging unit 30 are expressed by the following equation (6), the coordinates (up, vp) of the projected pixel can be expressed by the following equation (7).

[0064] As a result, based on the above equations (3) and (7), the coordinates (up, vp) of the projected pixel corresponding to the coordinates (uc, vc) of the imaged pixel on the image plane F1 can be obtained as any point on the epipolar line Le on the projection plane F2 shown in Figure 13.

[0065] Since the measurement distance range of the three-dimensional measurement apparatus 10 is determined by design, Zc in spatial coordinates is a value between the minimum measurement distance Zmin and the maximum measurement distance Zmax. Therefore, the line segment of the epipolar line Le on the projection plane F2 can be determined as a line segment whose both ends are the coordinates (up_Zmin, vp_Zmin) of the projected pixel on the projection plane F2 onto which the spatial coordinates (Xc, Yc, Zmin) are mapped, and the coordinates (up_Zmax, vp_Zmax) of the projected pixel on the projection plane F2 onto which the spatial coordinates (Xc, Yc, Zmax) are mapped, as shown in FIG.

[0066] The coordinates (up, vp) of the projection pixel that projects onto the coordinates (uc, vc) of the central imaging pixel of the first imaging pixel group lie on the segment of the epipolar line Le determined in this manner. Therefore, in the process for identifying the projection pixel group in this embodiment, as shown in FIG. 14B , a region in which the segment of the epipolar line Le is thickened using a predetermined value (e.g., the region Po indicated by the two-dot chain line in FIG. 14B ) is identified as the projection pixel group of the projection unit 20, whose projection region is the region captured by the first imaging pixel group. The predetermined value can be set, for example, to a value corresponding to the opening diameter Dp of the projection aperture 21 for the projection plane F2.

[0067] The above-described process for identifying a projected pixel group identifies a projected pixel group of the projection unit 20, whose projection area is the imaging area of ​​the first imaging pixel group. Compared to a case where the minimum measurement distance Zmin and the maximum measurement distance Zmax are not taken into consideration, the projected pixel group can be identified more accurately. In other words, the area of ​​the identified projected pixel group can be narrowed. This makes it possible to prevent a decrease in measurement accuracy due to the absence of event data output from the identified projected pixel group that is set to the off state.

[0068] In the process for identifying the projection pixel group, the projection pixel group is identified by a region in which the line segments of the epipolar line Le for the coordinates (uc, vc) of the imaging pixel are thickened using the predetermined value. However, the projection pixel group is not limited to this, and may be identified by other methods using the epipolar line Le. For example, as shown in FIG. 15A , multiple epipolar line Le segments are obtained for several imaging pixels on the edge of the first imaging pixel group. Then, as shown in FIG. 15B , a region including the multiple epipolar line Le segments may be identified as the projection pixel group. In FIG. 15A , the opening diameter Dp of the projection aperture 21 for the projection plane F2 is used to determine relative positions (D / 2 cos θ, D / 2 sin θ) with respect to the center of the first imaging pixel group. The several imaging pixels are pixels at the relative positions. However, the epipolar line Le is not limited to being obtained for several imaging pixels on the edge of the first imaging pixel group in this manner. For example, the epipolar line Le may be determined for all or a predetermined part of the first imaging pixel group.

[0069] Also, in other embodiments, the projected pixel group may be identified based on an epipolar line that converts the coordinates of the captured image into the coordinates of the projected pixels, as in this embodiment.

[0070] The present disclosure is not limited to the above-described embodiments, and may be modified as follows, for example.

[0071] (1) The three-dimensional measurement processing according to the embodiment of the present disclosure is not limited to being applied to three-dimensional measurement processing using an event camera employing a phase shift method. The three-dimensional measurement processing according to the embodiment of the present disclosure may be applied to other measurement methods (for example, three-dimensional measurement processing using an event camera employing a light-section method).

[0072] In a conventional stripe pattern for light-section imaging, multiple stripe regions whose luminance changes at a predetermined rate in a first direction and whose luminance does not change in a second direction perpendicular to the first direction are arranged along the first direction. The imaging unit 30 images the surface onto which this stripe pattern is projected. Bright lines formed by connecting pixel positions identified from negative event data output during the same time period are imaged as vertical lines for each stripe region. The bright lines for each stripe region move from left to right within a unit time. The movement of the bright lines is imaged as a video. These vertically moving bright lines (i.e., lines formed by connecting pixel positions identified from negative event data) can be used as line-shaped laser light used in light-section imaging.

[0073] Therefore, even in the case of a stripe pattern for the light-section method, the projection pixel group of the projection unit, whose projection area is the image capture area of ​​the first imaging pixel group detected as described above, is identified. A stripe pattern in which the identified projection pixel group is in an off state is projected. This makes it possible to improve the measurement accuracy of the measurement object R1, which is prone to specular reflection.

[0074] (2) The three-dimensional measuring device 10 is not limited to being attached to the hand of a robot and moving to measure the three-dimensional shape of a measurement target that moves relative to the robot. For example, the three-dimensional measuring device 10 may be fixed and measure the three-dimensional shape of a measurement target that moves on a conveyance line.

[0075] (3) The projection unit 20 and the image capturing unit 30 may be independent of the measurement unit 40 in the three-dimensional measurement apparatus 10. In this case, the measurement unit 40 may be an information processing terminal capable of wireless or wired communication with the projection unit 20 and the image capturing unit 30.

Claims

1. A three-dimensional measuring device comprising: a projection unit which projects a prescribed stripe pattern onto a measurement target area; an imaging unit which images a measurement target object placed in the measurement target area onto which the prescribed stripe pattern is projected, and outputs event data including two-dimensional point data identifying the positions of imaging pixels whose luminance has changed when light is received; a measurement unit which measures the three-dimensional shape of the measurement target object imaged by the imaging unit; and a projection control unit which controls the projection unit, wherein the measurement unit comprises: a determination unit which determines whether an image generated from the event data includes an area where specular reflection occurs; a detection unit which detects an imaging pixel group of the imaging unit which has imaged the area where specular reflection occurs in the image determined by the determination unit to include the area where specular reflection occurs; and an identification unit which identifies a projection pixel group of the projection unit as a projection area formed by the imaging pixel group detected by the detection unit, wherein the projection control unit projects the prescribed stripe pattern in which the projection pixel group identified by the identification unit is in an off state when it is determined by the determination unit that the area where specular reflection occurs.

2. The three-dimensional measuring device of claim 1, wherein the predetermined stripe pattern is generated so that within a unit time, a determination timing, which is the turning-off timing of a group of projection pixels whose projection area is outside the imaging area of ​​the imaging unit, is earlier than the turning-off timing of projection pixels whose projection area is the imaging area of ​​the imaging unit, and the detection unit detects the group of imaging pixels from which the event data is output in accordance with the determination timing as the group of imaging pixels that has captured the area where specular reflection occurs.

3. The three-dimensional measuring device according to claim 1, wherein the detection unit detects a group of imaging pixels capturing an image of an area where specular reflection occurs, the area being centered on an imaging pixel with the largest event amount.

4. The three-dimensional measuring device according to claim 1, wherein the identification unit identifies the projection pixel group based on a line segment on an epipolar line for the coordinates of at least one imaging pixel of the imaging pixel group detected by the detection unit, the epipolar line being a straight line that converts the coordinates of the at least one imaging pixel into the coordinates of a projection pixel, and the line segment on the epipolar line is a line segment whose two ends are the coordinates of projection pixels on the epipolar line that correspond to the minimum measurement distance and the maximum measurement distance, respectively.

Images