Three-dimensional measurement device

Abstract

The present invention improves distance accuracy by suppressing the influence of background light noise in a three-dimensional measurement device based on event data. A light source unit (10) irradiates an object (OB) with structured light including pulse light having a frequency different from that of the object (OB). An imaging unit (20) captures an image of the object (OB) that is irradiated with the structured light, and outputs event data for identifying the position of a pixel having a change in brightness. An event signal processing unit (40) calculates an event occurrence probability from the number of times of occurrence of the event with respect to the number of times of pulse light emission. An image generation unit (50) uses the event occurrence probability to determine a zone boundary used for spatial coding for distance image generation.

Description

3D measurement device

[0001] The present disclosure relates to a technique for optically measuring the three-dimensional shape of an object, and in particular to three-dimensional measurement using an event camera.

[0002] The event camera is a brightness value differential output camera developed based on the retinal structure of living organisms, and is configured to detect changes in brightness for each pixel and output the coordinates, time, and polarity of the brightness change (event data). Unlike conventional cameras, the event camera does not output pixel information with no brightness change, so it is effective in reducing data communication volume and image processing load.

[0003] Patent Literature 1 discloses a three-dimensional measuring device that measures the three-dimensional shape of a measurement object or information indicating that shape based on event data. In this three-dimensional measuring device, a predetermined stripe pattern is optically projected from a projection unit, and an imaging unit captures an image of the object onto which the stripe pattern is projected. The imaging unit includes an imaging element that outputs event data and generates a captured image from this event data. The measurement unit calculates brightness information based on the time difference between the output of event data for a positive brightness change and the output of event data for a negative brightness change.

[0004] International Publication No. 2021 / 085419

[0005] However, event cameras have a problem in that the probability of successful event detection decreases when capturing scenes with a high amount of background light. This is because signal fluctuations caused by random noise such as optical shot noise are mistakenly detected as events. In other words, background light noise makes event detection uncertain in event cameras.

[0006] For this reason, in the three-dimensional measurement device based on event data disclosed in Patent Document 1, the accuracy of the output time difference may decrease and the distance accuracy may decrease due to the occurrence of multiple events with positive and negative brightness changes due to variations in the amount of background light.

[0007] In view of the above, an object of the present disclosure is to suppress the influence of background light noise and improve distance accuracy in a three-dimensional measurement device based on event data.

[0008] A three-dimensional measuring device according to one aspect of the present disclosure includes a light source unit that irradiates an object with structured light including pulsed light of different frequencies; an imaging unit that images the object irradiated with the structured light and outputs event data that identifies the position of a pixel where a change in brightness has occurred; an event signal processing unit that calculates, for each imaging region, an event occurrence probability from the number of event occurrences at which the imaging unit detects an event relative to the number of pulsed emissions from the light source unit; and an image generation unit that generates a distance image of the object by performing spatial encoding using the output of the imaging unit and determines zone boundaries to be used for spatial encoding using the event occurrence probability.

[0009] According to the present disclosure, in a three-dimensional measurement device based on event data, it is possible to suppress the influence of background light noise and improve distance accuracy.

[0010] An example of the configuration of a three-dimensional measurement device according to an embodiment. An example of structured light emitted by a light source unit. A diagram showing the time change in the cumulative number of events when structured light is emitted, and zone boundaries for determining code values. A diagram for explaining the influence of background light on event detection. A diagram for explaining the influence of background light on event detection. (a) and (b) are diagrams for explaining the event occurrence probability. (a) and (b) are examples of determining zone boundaries using the event occurrence probability. (a) and (b) are examples of pulse emission patterns.

[0011] (Summary) A three-dimensional measuring device according to an aspect of the present disclosure includes a light source unit that irradiates an object with structured light including pulsed light of different frequencies; an imaging unit that images the object irradiated with the structured light and outputs event data that identifies the position of a pixel where a luminance change has occurred; an event signal processing unit that calculates, for each imaging region, an event occurrence probability from the number of event occurrences detected by the imaging unit relative to the number of pulsed emissions from the light source unit; and an image generation unit that generates a distance image of the object by performing spatial encoding using the output of the imaging unit and determines zone boundaries to be used for spatial encoding using the event occurrence probability.

[0012] With this configuration, the light source unit irradiates an object with structured light including pulsed light of different frequencies, and the imaging unit captures an image of the object irradiated with the structured light and outputs event data. The event signal processing unit calculates an event occurrence probability based on the number of event occurrences relative to the number of pulse emissions. This event occurrence probability varies depending on the intensity and variability of background light. The image generation unit determines zone boundaries to be used for spatial encoding using the event occurrence probability. This makes it possible to reflect the effects of the intensity and variability of background light in spatial encoding using structured light. Therefore, the influence of background light noise can be suppressed and distance accuracy can be improved in a 3D measurement device.

[0013] In the three-dimensional measuring apparatus according to the above aspect, the image generating section may generate a brightness image using the event occurrence probability.

[0014] This allows a luminance image to be generated using the calculated event occurrence probability.

[0015] In the three-dimensional measuring apparatus according to the above aspect, the light source unit may be configured such that, during a period of light emission, a section for emitting pulsed light and a section for emitting continuous light are set.

[0016] This makes it possible to detect events caused by object movement, such as vibrations, in the section where continuous light is irradiated.

[0017] In the three-dimensional measuring device according to the above aspect, the light source unit may repeatedly perform an operation of irradiating structured light, and the event signal processing unit may calculate the event occurrence probability in each imaging region each time the light source unit performs the operation.

[0018] This allows the zone boundaries used for spatial coding to be determined more accurately, making it possible to more appropriately reflect the effects of background light intensity and variations in spatial coding using structured light.

[0019] (Embodiments) Hereinafter, embodiments will be specifically described with reference to the drawings.

[0020] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0021] FIG. 1 is a block diagram showing an example configuration of a three-dimensional measuring device according to an embodiment. In FIG. 1, a light source unit 10 irradiates an object OB with structured light for encoding space. This structured light includes pulsed light of different frequencies. An event camera 20 corresponding to the imaging unit images the object OB irradiated with the structured light and outputs event data specifying the position of a pixel where a luminance change has occurred. The event data includes, for example, the coordinates of the pixel where the luminance change occurred, the time when the luminance change occurred, and the polarity of the luminance change. A control unit 30 controls the light emission operation of the light source unit 10 and the imaging operation of the event camera 20.

[0022] The event signal processing unit 40 receives and processes event data output from the event camera 20. The event signal processing unit 40 includes an event occurrence probability calculation unit 41 that calculates an event occurrence probability using the number of event occurrences detected by the event camera 20 and the number of pulsed emissions from the light source unit 10. The definition and usage of the event occurrence probability will be described later. The event occurrence probability calculation unit 41 calculates the event occurrence probability for each imaging area.

[0023] The image generation unit 50 receives the output of the event signal processing unit 40 and performs processing. The image generation unit 50 includes a distance image generation unit 51, a frequency / vibration image generation unit 52, and a luminance image generation unit 53. In this embodiment, the image generation unit 50 can generate a distance image, a luminance image, a frequency image, and a vibration image. The output unit 60 is, for example, a display, and outputs the distance image and the like generated by the image generation unit 50.

[0024] Fig. 2 is a diagram showing an example of structured light emitted by the light source unit 10. The structured light in Fig. 2 emits multiple types of linear pulsed light with different frequencies. In Fig. 2, a position irradiated with high-frequency light is designated as code 1, a position irradiated with low-frequency light is designated as code 2, and a position where no light is irradiated is designated as code 2. Note that the pulsed light with different frequencies does not have to be linear.

[0025] Figure 3 is a graph showing the change in the cumulative number of events over time when irradiating the structured light shown in Figure 2. Assuming that there is no influence of background light noise, event data is detected every time the intensity of the irradiated light changes. As a result, the cumulative number of events increases over time. Also, the cumulative number of events is higher for high-frequency light and lower for low-frequency light.

[0026] Then, zone boundaries for determining code values ​​are set on the graph in Figure 3. In Figure 3, the area from the first zone boundary to the second zone boundary is code 1. The area from the second zone boundary to the third zone boundary is code 2. In accordance with the setting of these zone boundaries, the code for the pixel position can be determined based on the cumulative number of events at a certain measurement time A.

[0027] The effect of background light on event detection will now be described with reference to FIGS.

[0028] The two images in the upper left of Figure 4 show cases where signal changes occur at two locations A and B in a state where there is no background signal (pitch black). The signal change amounts at locations A and B are different, with the signal change amount at location A being greater than that at location B. The graph at the bottom of Figure 4 shows the time changes in luminance values ​​(input signal amount) at locations A and B. At the timing when the signal change occurred, the luminance change amount at both locations A and B exceeded the event threshold, so events were detected at both locations A and B.

[0029] The two images in the upper left of Figure 5 show cases where signal changes occur at two locations A and B when there is a background signal amount (slightly bright). The signal change amounts at locations A and B are different, with location A having a larger signal change amount than location B. In this case, for example, an event is detected at location A, but at location B, the amount of brightness change is small, so it is unclear whether the event threshold will be exceeded and therefore whether an event will be detected. Furthermore, as shown in the graph on the lower right, when there is a background signal amount and it is bright, there are areas where an event will occur probabilistically.

[0030] Therefore, in this disclosure, the concept of event occurrence probability is introduced to utilize the influence of background light in a negative way. The event occurrence probability is defined by the following equation.

[0031]

[0032] Figure 6 shows an image of the event occurrence probability. In Figure 6, an ON event (black arrow) indicates an event that occurs when the signal level increases, and an OFF event (white arrow) indicates an event that occurs when the signal level decreases.

[0033] Figure 6(a) shows a case where the signal amount of background light is small and the variation is small. In the example of Figure 6(a), when pulsed light is irradiated, an ON event always occurs when the pulsed light is generated and the signal increases, and an OFF event always occurs when the pulsed light disappears and the signal decreases. That is, in the example of Figure 6(a), the number of emitted pulses is 4, and the cumulative number of event occurrences is 8. When substituted into the above definition formula, the event occurrence probability becomes 100%.

[0034] Figure 6(b) shows a case where the signal amount of background light is large and varies greatly. In the example of Figure 6(b), when irradiating pulsed light, there are cases where an ON event cannot be detected even when the pulsed light is generated and the signal increases, and cases where an OFF event cannot be detected even when the pulsed light disappears and the signal decreases. That is, in the example of Figure 6(b), the number of emitted pulses is 4, and the cumulative number of event occurrences is 4. Substituting this into the above definition formula, the event occurrence probability is 50%.

[0035] In the present disclosure, the calculated event occurrence probability is used to generate a distance image. Specifically, for example, when spatial encoding is performed using structured light including pulsed light of different frequencies, the calculated event occurrence probability is used to determine the zone boundaries shown in FIG. 3 .

[0036] 7(a) and 7(b) show an example of determining zone boundaries using the event occurrence probability. Similar to the graph in Fig. 3, the graphs in Fig. 7(a) and 7(b) show the relationship between time and the cumulative number of events when irradiating with structured light. Fig. 7(a) shows the case where the event occurrence probability is 50%, and Fig. 7(b) shows the case where the event occurrence probability is 80%.

[0037] Assume now that the number of light emission pulses of the low-frequency light is 50 times per second and the number of light emission pulses of the high-frequency light is 100 times per second. In this case, as shown in Figure 7(a), if the event occurrence probability is 50%, the cumulative number of events after 0.5 seconds will be 25 for the low-frequency light and 50 for the high-frequency light. In this case, the second zone boundary is set as a straight line that passes through 37.5 times, which is the average of 25 times and 50 times, after 0.5 seconds.

[0038] 7B, when the event occurrence probability is 80%, the cumulative number of events after 0.5 seconds is 40 for low-frequency light and 80 for high-frequency light. In this case, the second zone boundary is set as a straight line that passes through 60, which is the average of 40 and 80, after 0.5 seconds.

[0039] In this way, by using the event occurrence probability to determine the zone boundaries used when spatially encoding by irradiating structured light containing pulsed light of different frequencies, it is possible to suppress the influence of background light and generate highly accurate distance images.

[0040] It is also possible to generate a luminance image using the event occurrence probability. For example, the luminance value is calculated for each pixel from the event occurrence probability using the following equation: Then, a luminance image is generated using the luminance value of each pixel.

[0041]

[0042] The above formula is an example when an 8-bit output sensor is used, and the numerator value "256" can be changed according to the number of bits of the output sensor.

[0043] In addition, although the above formula linearly converts the event occurrence probability and brightness value, this conversion does not necessarily have to be linear. For example, to make the brightness value appear clearer, gamma conversion and gain adjustment, which are performed in normal image processing, can be performed in later processing.

[0044] FIG. 8 shows an example of a pulsed light emission pattern, where (a) shows pulsed light emission in a pixel region corresponding to a low-frequency code, and (b) shows pulsed light emission in a pixel region corresponding to a high-frequency code. As shown in FIG. 8 , during the period in which the light source unit 10 emits light, it is preferable to set a section in which continuous wave (CW) light is emitted between pulsed light emission sections. Events caused by object movement, such as vibration, can be detected during this CW emission section. Furthermore, vibrations of textureless objects can also be detected by emitting light in a pattern with different light intensities. That is, a frequency image can be generated from the number of events occurring during the CW emission section. Furthermore, an amplitude image can be generated by referencing distance values. Since the generation of frequency images and amplitude images can be easily achieved using known techniques, a detailed description thereof will be omitted here.

[0045] Alternatively, the light source unit 10 may repeatedly perform the operation of irradiating structured light, and the event signal processing unit 40 may calculate the event occurrence probability in each imaging region each time the light source unit 10 performs the operation of irradiating structured light. This allows the zone boundaries used in spatial coding to be determined with greater accuracy, making it possible to more appropriately reflect the effects of the intensity and variations of background light in spatial coding using structured light.

[0046] As described above, in this embodiment, the light source unit 10 irradiates the object OB with structured light including pulsed light of different frequencies, and the event camera 20 captures an image of the object OB irradiated with the structured light and outputs event data. The event signal processing unit 40 calculates an event occurrence probability based on the number of event occurrences relative to the number of pulsed emissions. This event occurrence probability varies depending on the intensity and variability of background light. The image generation unit 50 uses the event occurrence probability to determine zone boundaries to be used for spatial encoding. This makes it possible to reflect the effects of the intensity and variability of background light in spatial encoding using structured light. Therefore, the influence of background light noise can be suppressed in the three-dimensional measurement device, thereby improving distance accuracy. Furthermore, the image generation unit 50 can generate a luminance image using the calculated event occurrence probability.

[0047] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0048] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0049] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0050] The present disclosure is useful for improving the accuracy of three-dimensional measurement, for example, because it can suppress the influence of background light noise and improve distance accuracy in a three-dimensional measurement device based on event data.

[0051] 10 Light source unit 20 Event camera (imaging unit) 40 Event signal processing unit 50 Image generation unit OB Object

Claims

1. A three-dimensional measurement device comprising: a light source unit that irradiates an object with structured light including pulsed light of different frequencies; an imaging unit that images the object irradiated with the structured light and outputs event data specifying the position of a pixel where a luminance change has occurred; an event signal processing unit that calculates, for each imaging area, an event occurrence probability from the number of event occurrences at which the imaging unit detects an event relative to the number of pulsed emissions from the light source unit; and an image generation unit that generates a distance image of the object by performing spatial encoding using the output of the imaging unit, and that determines zone boundaries to be used for spatial encoding using the event occurrence probability.

2. A three-dimensional measuring device according to claim 1, wherein the image generating section generates a brightness image using the event occurrence probability.

3. A three-dimensional measuring device according to claim 1, wherein the light source unit has a period during which it emits light, during which it sets a section for irradiating pulsed light and a section for irradiating continuous light.

4. A three-dimensional measuring device according to claim 1, wherein the light source unit repeatedly executes an operation of irradiating structured light, and the event signal processing unit calculates the event occurrence probability in each imaging region each time the light source unit executes the operation.

Images