Three-dimensional measurement device, three-dimensional measurement method, and program

Abstract

To improve accuracy of measurement in three-dimensional measurement using an event vision base camera.SOLUTION: An irradiation device irradiates a measurement object with an electromagnetic wave and scans the electromagnetic wave in a predetermined cycle. An event vision base camera detects the scattering of the electromagnetic wave, with which the irradiation device irradiates, scattered by the measurement object. Event time series is time series generated on the basis of an output of the event vision base camera and is time series indicating the presence / absence of ignition which is a phenomenon in which an event detection signal indicating the detection of a luminance change in a pixel included in the event vision base camera is generated. A three-dimensional measuring device acquires the event time series for each pixel included in the event vision base camera, acquires correlation intensity time series indicating time change in intensity of cross correlation between the event time series and a template function which is a predetermined function of the cycle for each pixel, estimates an angle or a position at which the irradiation device irradiates with the electromagnetic wave detected by each pixel on the basis of the correlation intensity time series, and estimates a three-dimensional shape of the measurement object on the basis of a result of estimation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a three-dimensional measurement device, a three-dimensional measurement method, and a program.

Background Art

[0002] As a method of three-dimensional measurement using a camera and a projector, there is an active stereo method. The camera and the projector are installed in the same direction, pattern illumination is projected, and the corresponding pixels on the projector are specified from the luminance values observed by the camera, thereby obtaining the parallax between the camera and the projector and calculating the three-dimensional position.

[0003] In the active stereo method, when the subject is translucent and transmits part of the light, the luminance values observed by the camera are a mixture of reflections from multiple layers, so the corresponding pixels on the projector cannot be specified. Therefore, a frame-based camera that captures one frame at a time may be used. A frame-based camera outputs the absolute amount of luminance for each frame. However, when using a frame-based camera, the pixels on the projector are lit one by one at a time, and by capturing one frame at a time, reflections from multiple layers can be identified, but it is necessary to capture as many times as the number of pixels.

[0004] There is an event-based vision camera, which is a camera equipped with an event-based vision sensor, as a camera of a different type from the frame-based camera. In an event-based vision camera, each pixel operates asynchronously and outputs the luminance change of each pixel with a time resolution of microseconds. Compared with the frame-based method, the event-based vision camera has the characteristics of high time resolution, wide dynamic range, and only changes can be acquired.

[0005] In three-dimensional measurement using an event-based vision camera, illumination by a projector is applied to a subject, and the reflected light is observed by the event-based vision camera. When it is assumed that the subject does not transmit light and there is no scattering due to the surrounding environment, if only one pixel of the projector irradiates light, the scattered light in a minute region on the corresponding subject surface is observed by the event-based vision camera, and an event occurs at the corresponding pixel. Note that the scattering due to the surrounding environment is scattering in which the scattered light generated when the illumination hits something unrelated to the observation target such as a wall or a floor does not affect the observation target.

[0006] In three-dimensional measurement using an event-based vision camera, scanning of the subject is performed by changing the illuminated pixels at a constant period. When performing such scanning, if the scanning speed is sufficiently low with respect to the time resolution of the event-based vision camera, events due to scattered light from the irradiation range at each time can be observed individually. When using a laser irradiation type projector using MEMS (Micro Electro Mechanical Systems) as the projector, even if all pixels are constantly illuminated, the laser irradiation position changes with time due to the driving of the MEMS, so the same observation can be performed.

[0007] And when the geometric relationship between the projector and the event-based vision camera is known, at each time, from the relationship between the illuminated pixel position, which is the position of the pixel illuminated in the projector, and the firing pixel position, which is the position of the pixel where an event has fired in the event-based vision camera, the three-dimensional position of the corresponding subject can be calculated. Note that in the event-based vision camera, for an event to fire at a certain pixel means that the amount of change in the luminance value at a certain pixel of the event-based vision camera exceeds a threshold value and an event detection signal is generated.

[0008] When not considering transmission or internal scattering, if the light reflected and scattered from a certain point on the subject causes an event to occur at one pixel on the event vision sensor, one event occurs per scanning period T of the projector at this pixel. In this case, the event ignition pixel position corresponds one-to-one to the illuminated pixel position of the projector, and the illuminated pixel position of the corresponding projector can be specified from the event ignition time at each event ignition pixel position. From this relationship, the three-dimensional position can be obtained by determining the parallax between the camera and the projector at the scattering point.

Summary of the Invention

Problems to be Solved by the Invention

[0009] However, when the subject transmits light and scattering occurs at each point on the surface and inside of the subject, events may occur at multiple pixels of the event-based vision camera in response to the illumination of a single pixel on the projector. As a result, for a single pixel of the event-based vision camera, multiple events may be observed during one period of the irradiation period due to light reaching the pixel from different reflection positions.

[0010] FIG. 13 is an explanatory diagram for explaining the occurrence of events at multiple pixels of the event-based vision camera and the observation of multiple events at a single pixel of the event-based vision camera. FIG. 13 shows two types of lasers with different irradiation timings, i.e., irradiation angles, and the paths from when each laser is irradiated until it reaches the camera. The parallax can be calculated from these event occurrence positions and timings, and the three-dimensional positions of the reflection positions on the front and back surfaces can be calculated from the illuminated pixel positions of the projector at the same timing.

[0011] By the way, in an event-based vision camera, many events are missed, noises occur, and there are delays in event firing due to insufficient bandwidth or insufficient light quantity. If an event is missed, it is impossible to calculate the three-dimensional position. If there is a noise event, noise will also occur in the restored three-dimensional information. In addition, if there is a delay in the event firing time, a deviation in the pixel correspondence relationship occurs between the event vision sensor and the projector, and the correct parallax cannot be obtained.

[0012] Particularly, the light reflected from the back of the subject often has insufficient light quantity because it passes through the front, so it is easily affected by such problems. Thus, in three-dimensional measurement using an event-based vision camera, the measurement accuracy may be poor. Note that a delay in the event firing time means a state in which the event firing time recorded in the event detection signal is shifted with respect to the time when the luminance change actually occurred. The time when the luminance change actually occurred is the time when the illumination is irradiated from the projector to the observation point or the time when the irradiation ends.

[0013] In view of the above circumstances, an object of the present invention is to provide a technique for improving the measurement accuracy in three-dimensional measurement using an event vision-based camera.

Means for Solving the Problems

[0014] One aspect of the present invention is a three-dimensional measurement device comprising: an irradiation device that irradiates a measurement object with electromagnetic waves and scans the electromagnetic waves at a predetermined period; an event vision-based camera that detects scattering of the electromagnetic waves irradiated by the irradiation device by the measurement object; an event time series acquisition unit that acquires, for each pixel included in the event vision-based camera, an event time series which is a time series generated based on the output of the event vision-based camera and indicates the presence or absence of ignition, which is a phenomenon in which an event detection signal indicating that a luminance change of a pixel included in the event vision-based camera has been detected occurs; an event correlation intensity acquisition unit that acquires, for each pixel, a correlation intensity time series indicating a temporal change in the strength of the cross-correlation between the event time series and a template function which is a predetermined function of the period; an irradiation situation estimation unit that estimates, based on the correlation intensity time series, the angle or position at which the electromagnetic waves detected by each pixel are irradiated from the irradiation device; and a shape estimation unit that estimates the three-dimensional shape of the measurement object based on the estimation result of the irradiation situation estimation unit.

[0015] One aspect of the present invention is a three-dimensional measurement device comprising: an irradiation device that irradiates a measurement object with electromagnetic waves and scans the electromagnetic waves at a predetermined period; an event vision-based camera that detects scattering of the electromagnetic waves irradiated by the irradiation device by the measurement object; an event time series acquisition unit that acquires, for each pixel included in the event vision-based camera, an event time series which is a time series generated based on the output of the event vision-based camera and indicates the presence or absence of ignition, which is a phenomenon in which an event detection signal indicating that a luminance change of a pixel included in the event vision-based camera has been detected occurs; a clustering for each ignition indicated by the event time series, a process of estimating a cluster in which the distance between clusters is the period based on the result of the clustering, and for the cluster estimated to have a distance between clusters of the period, a process of estimating the angle or position at which the electromagnetic waves detected by each pixel are irradiated from the irradiation device based on the values of each element of the cluster; a cluster estimation unit that executes the above processes; and a shape estimation unit that estimates the three-dimensional shape of the measurement object based on the estimation result of the cluster estimation unit.

[0016] One aspect of the present invention is a three-dimensional measurement method including: an event time series acquisition step of acquiring, for each pixel included in an event vision-based camera, an event time series which is a time series indicating the presence or absence of ignition, the event time series being generated based on an output of the event vision-based camera that detects scattering of the electromagnetic wave irradiated by an irradiation device that irradiates the measurement target with the electromagnetic wave and scans the electromagnetic wave at a predetermined period, and is a time series indicating a phenomenon in which an event detection signal indicating that a luminance change of a pixel included in the event vision-based camera is detected occurs; an event correlation intensity acquisition step of acquiring, for each pixel, a correlation intensity time series indicating a temporal change in the strength of the cross-correlation between the event time series and a template function which is a predetermined function of the period; an irradiation situation estimation step of estimating, based on the correlation intensity time series, an angle or position at which the electromagnetic wave detected by each pixel is irradiated from the irradiation device; and a shape estimation step of estimating a three-dimensional shape of the measurement target based on a result of the estimation in the irradiation estimation step.

[0017] One aspect of the present invention is a three-dimensional measurement method including: an event time series acquisition step of acquiring, for each pixel included in an event vision-based camera, an event time series which is a time series indicating the presence or absence of ignition, the event time series being generated based on an output of the event vision-based camera that detects scattering of the electromagnetic wave irradiated by an irradiation device that irradiates the measurement target with the electromagnetic wave and scans the electromagnetic wave at a predetermined period, and is a time series indicating a phenomenon in which an event detection signal indicating that a luminance change of a pixel included in the event vision-based camera is detected occurs; a clustering for each ignition indicated by the event time series; a process of estimating a cluster in which a distance between clusters is the period based on a result of the clustering; a process of estimating, for the cluster estimated to have a distance between clusters of the period, an angle or position at which the electromagnetic wave detected by each pixel is irradiated from the irradiation device based on values of each element of the cluster; a cluster estimation step of executing the processes; and a shape estimation unit of estimating a three-dimensional shape of the measurement target based on a result of the estimation in the cluster estimation step.

[0018] One aspect of the present invention is a program for causing a computer to function as the above-described three-dimensional measurement device.

Advantages of the Invention

[0019] According to the present invention, it becomes possible to improve the measurement accuracy in three-dimensional measurement using an event vision-based camera.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0021] (Embodiment) Using FIGS. 1 to 6, an overview of the three-dimensional measurement system 100 of the embodiment will be described. The three-dimensional measurement system 100 is a system that irradiates an irradiation target 9 with electromagnetic waves and estimates the three-dimensional shape of the irradiation target 9 based on the scattering thereof. Note that, in the description of the three-dimensional measurement system 100, scattering is used as a term indicating a phenomenon in which the propagation direction of electromagnetic waves changes. Therefore, reflection is described as a type of scattering in the three-dimensional measurement system 100. First, an overview of the processing executed by the three-dimensional measurement system 100 will be described by taking the case where the electromagnetic wave is a laser as an example.

[0022] <Overview> FIG. 1 is an explanatory diagram for explaining an overview of the three-dimensional measurement system 100 of the embodiment. The three-dimensional measurement system 100 includes an irradiation device 1, an event vision-based camera 2, and a three-dimensional measurement device 3. Details of the irradiation device 1, the event vision-based camera 2, and the three-dimensional measurement device 3 will be described later, but the irradiation device 1 is a device that irradiates a laser. The event vision-based camera 2 detects the laser scattered by the irradiation target 9 and outputs a signal indicating the detection (hereinafter referred to as an "event detection signal"). The three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on the event detection signal.

[0023] FIG. 2 is a diagram showing an example of the state of one scan of the laser irradiated by the irradiation device 1 in the embodiment. As shown in FIG. 2, the irradiation device 1 irradiates the laser in a scanning manner. That is, the irradiation device 1 irradiates the laser while changing the irradiation angle, for example.

[0024] FIG. 3 is a diagram showing an example of the light-receiving surface of the event vision-based camera 2. The event vision-based camera 2 includes a light-receiving unit 21. The light-receiving unit 21 exists on the light-receiving surface of the event vision-based camera 2. The light-receiving unit 21 includes a plurality of event elements 210 arranged in an array. The event element 210 is an element that converts light into an electrical signal by photoelectric conversion. The electrical signal generated by the conversion is specifically an event detection signal.

[0025] The event detection signal is input to the three-dimensional measurement device 3. FIG. 4 is a diagram showing an example of the timing at which the three-dimensional measurement device 3 in the embodiment acquires the event detection signal. More specifically, it is a diagram showing an example of the timing at which the three-dimensional measurement device 3 acquires the event detection signal output by one event element 210. The horizontal axis of FIG. 4 represents time, and the vertical axis represents intensity. The peak in FIG. 4 represents the event detection signal. The three-dimensional measurement device 3 obtains information indicating the timing of the occurrence of the event detection signal for each event element 210 as shown in FIG. 4.

[0026] In the example of FIG. 4, the event detection signal has two peaks during the time interval T. The time interval T is the time for one scan of the laser. Therefore, the laser is scanned at a period T. Hereinafter, one of the two peaks existing during the time interval T in FIG. 4 will be referred to as the left peak, and the other will be referred to as the right peak.

[0027] The left peak and the right peak are generated by lasers, for example, with different irradiation timings. Specifically, different irradiation timings mean different irradiation phases. The phase is the elapsed time since the start of each unit period. That is, when the start time of the m-th (m is an integer greater than or equal to 1) scan is defined as T0_m, the elapsed time from the start time T0_m in one scan with the start time T0_m is the irradiation phase. The unit period is a period with a length of the period T. Since the operation of the laser is repeated at the period T, in FIG. 4, the left peak repeatedly appears at substantially the same timing at the time interval T. Also, since the operation of the laser is repeated at the period T, in FIG. 4, the right peak repeatedly appears at substantially the same timing at the time interval T.

[0028] In the example of FIG. 4, there are two types of peaks, the left peak and the right peak, but the number of peaks that appear within the time interval T and repeatedly appear at substantially the same period of the period T is not necessarily two. The number of peaks that appear within the time interval T and repeatedly appear at substantially the same period of the period T may be N (N is an integer greater than or equal to 1).

[0029] The three-dimensional measuring device 3 obtains the strength of the cross-correlation with a predetermined function (hereinafter referred to as "template function") of a predetermined period T, for example, as shown by a graph in FIG. 5, for data indicating that N types of peaks repeatedly appear at substantially the same period of the period T as shown in FIG. 4. An example of a graph showing the strength of the cross-correlation is FIG. 6. Specifically, obtaining the strength of the cross-correlation means performing a convolution integral.

[0030] FIG. 5 is a diagram showing an example of the template function in the embodiment. The horizontal axis of FIG. 5 indicates time, and the vertical axis of FIG. 5 indicates intensity. The function representing the graph of FIG. 5 is specifically a function indicating that the Gaussian function repeatedly appears at the period T. Hereinafter, for the sake of simplicity of explanation, the three-dimensional measurement system 100 will be described by taking the case where the template function is a function indicating that the Gaussian function repeatedly appears at the period T as an example.

[0031] FIG. 6 is a diagram showing an example of a graph indicating the strength of mutual correlation in an embodiment. The vertical axis of FIG. 6 indicates the strength of mutual correlation, and the horizontal axis represents the time difference between the peak of the Gaussian distribution in the unit period and the peak of the event detection signal input to the three-dimensional measurement device 3 in the unit period including the Gaussian distribution. Therefore, the horizontal axis is also an axis indicating the phase.

[0032] Actually, there is variation in the latency, which is the time from when the laser enters the event vision-based camera 2 until the event detection signal is output from the event vision-based camera 2. One of the causes of the variation is, for example, that the probability of the event vision-based camera 2 converting electromagnetic waves into electrons follows quantum mechanics such as Fermi's golden rule in photoelectric conversion. Another cause of the variation is, for example, that there is a limit to the amount of event detection signals that the event vision-based camera 2 can output per unit time. Due to this limit, even if many electrons are excited, the event detection signals are output little by little, and as a result, the latency varies.

[0033] The distribution showing such latency variation is represented by, for example, a Gaussian distribution. For the sake of simplicity in the following explanation, the case where the distribution showing the latency variation is a Gaussian distribution will be taken as an example. When the latency of the event detection signal output from the event vision-based camera 2 varies in a Gaussian distribution, the timing at which the event detection signal enters the three-dimensional measurement device 3 also varies in a Gaussian distribution.

[0034] Therefore, the probability that the three-dimensional measurement device 3 acquires an event detection signal is likely to be a probability in which a Gaussian distribution is repeated at a period T, as shown in FIG. 5, for example. As is clear from the description so far, the position of the peak of the Gaussian distribution in each unit period is the median of the latency variation. Note that the position of the peak of the Gaussian distribution is located, for example, at a time when the time difference from the start time of the unit period is the time when the probability of inputting an event detection signal to the three-dimensional measurement device 3 is the highest. Although details will be described later, the shape of the Gaussian distribution may depend not only on the latency but also on the amount of noise.

[0035] By the way, as described above, the process of obtaining the strength of the cross-correlation is specifically a convolution integral. Therefore, the process of obtaining the strength of the cross-correlation can express a phenomenon with a high appearance frequency more prominently than a phenomenon with a low appearance frequency. For example, consider a case where a peak that does not have a period T and appears only once in the graph of FIG. 4 is mixed outside the range of the Gaussian distribution shown by the template function.

[0036] Even if such a situation occurs, by obtaining the strength of the cross-correlation with the template function, it is possible to obtain a result of emphasizing the peak that appears at the period T rather than the peak that appears only once. Since the period T is the period of the laser scanning, the peak that appears at the period T is specifically N types of event detection signals. Note that specifically, emphasizing means obtaining a result with a stronger correlation intensity. Note that a peak that does not have a period T and appears only once is an example of noise.

[0037] Therefore, the process of obtaining the strength of the cross-correlation is a process of extracting an event detection signal generated due to laser scanning.

[0038] Based on the result indicating the strength of the cross-correlation thus obtained, the three-dimensional measurement device 3 obtains the phase of a peak that satisfies an intensity condition, which is a predetermined condition regarding the strength of the cross-correlation, among the peaks that appear within the unit period. Although details of an example of the intensity condition will be described later, for example, it is a condition that the strength of the cross-correlation is equal to or greater than a threshold value.

[0039] As described above, by obtaining the strength of the cross-correlation, N types of event detection signals, which are peaks appearing at the period T, are extracted. Then, as shown in FIG. 6, the information indicating the strength of the cross-correlation is, specifically, the information indicating the timings of the N types of event detection signals in the unit period. Therefore, by obtaining the strength of the cross-correlation, for each of the N types of event detection signals, information indicating the phase of the laser irradiation that caused it can be obtained.

[0040] Incidentally, when the irradiation target 9 is composed of, for example, an N-layer semi-transparent substance, the N types of event detection signals respectively correspond one-to-one to any one of the N layers. Corresponding to any one of the layers means that the nth (n is an integer from 1 to N) event detection signal among the N types of event detection signals is an event detection signal generated due to the laser scattered by the nth layer among the N layers.

[0041] The reason why such a thing can be said will be explained. A plurality of lasers with different irradiation phases can be incident on one event element 210. However, if the layer where the laser is reflected is the same regardless of the irradiation phase, according to the principle that the reflection angle is proportional to the incident angle, the lasers irradiated with different phases do not enter the same event element 210. Therefore, each of the N types of event detection signals output by one event element 210 is a signal generated due to the laser reflected by different layers.

[0042] Incidentally, the irradiation target 9 does not necessarily have to be composed of an N-layer semi-transparent substance. The irradiation target 9 may be a substance having N sites (hereinafter referred to as "electromagnetic wave scattering sites") that scatter lasers instead of N layers. In such a case, each of the N types of event detection signals is an event detection signal generated due to the laser scattered by mutually different electromagnetic wave scattering sites. That is, the N electromagnetic wave scattering sites and the N types of event detection signals correspond one-to-one.

[0043] In the three-dimensional measurement system 100, in this way, for each event element 210, the phase of the laser irradiation that generates various types of event detection signals is estimated, and based on the estimated result, the three-dimensional shape of the irradiation target 9 is estimated. Here, the description of the outline of the three-dimensional measurement system 100 ends.

[0044] <More detailed explanation> The three-dimensional measurement system 100 will be described again with reference to FIGS. 1 to 10 including FIGS. 1 to 6. As described above, the three-dimensional measurement system 100 includes an irradiation device 1, an event vision-based camera 2, and a three-dimensional measurement device 3. The three-dimensional measurement system 100 performs three-dimensional measurement on the irradiation target 9. Performing three-dimensional measurement on the irradiation target 9 means estimating the three-dimensional shape of the irradiation target 9. As will be described later, the three-dimensional shape is estimated by the three-dimensional measurement device 3. Therefore, the irradiation target 9 is the measurement target of the three-dimensional measurement system 100 or the three-dimensional measurement device 3.

[0045] The irradiation device 1 irradiates the irradiation target 9 with a laser and scans the laser at a predetermined period T. That is, the irradiation device 1 repeats the process of finishing one scan of the irradiation target 9 when the period T elapses.

[0046] The irradiation target 9 can be anything as long as it can scatter the laser irradiated by the irradiation device 1 at a plurality of positions. That is, the irradiation target 9 can be anything as long as it has a plurality of positions with a predetermined reflectivity greater than 0 and less than 1 with respect to the frequency of the laser irradiated by the irradiation device 1. The irradiation target 9 is, for example, an N-layer scatterer (N is an integer of 1 or more). Each layer of the N layers can be anything as long as it has a predetermined reflectivity greater than 0 and less than 1. Since the Kramers-Kronig relation generally holds in substances, scattering will always occur unless it is a perfect absorber.

[0047] A more detailed description will be given with respect to FIG. 2. More specifically, FIG. 2 shows the temporal change of the position on the camera where the laser irradiated by the irradiation device 1 reaches, in a scene where the laser is directly received by a camera arranged without a scatterer between the irradiation device 1. The direction of the vertical axis and the horizontal axis in FIG. 2 are two directions orthogonal to each other in real space. Point D1 indicates the position on the camera when the laser irradiated at the start time of one period reaches the camera. The direction of the arrow in FIG. 2 indicates the position of the laser reaching the camera, which moves as time elapses. Such a change occurs because the angle at which the irradiation device 1 irradiates the laser changes with time.

[0048] In this way, the angle at which the irradiation device 1 irradiates the laser and the irradiation phase have a one-to-one relationship. Note that the start time of the scan is the start time of the unit period, and the end time of the scan is the end time of the unit period.

[0049] FIG. 1 shows two types of lasers with different irradiation timings, that is, irradiation angles, of the laser irradiated by the irradiation device 1, and the paths from when each laser is irradiated until it reaches the event vision-based camera 2. That is, FIG. 1 shows the paths of the laser irradiated in the first phase and the laser irradiated in the second phase different from the first phase.

[0050] Three optical paths are shown as the optical paths of the laser in FIG. 1. The first optical path is an optical path that reaches the part R1 of the irradiation target 9 from the irradiation device 1, is scattered at the part R1, and then reaches the event vision-based camera 2. The second optical path is an optical path that reaches the part R1 of the irradiation target 9 from the irradiation device 1, passes through the part R1 and reaches the part R2 of the irradiation target 9, is scattered at the part R2, passes through the part R3 of the irradiation target 9, and then reaches the event vision-based camera 2. The third optical path is an optical path that reaches the part R3 of the irradiation target 9 from the irradiation device 1, is scattered at the part R3, and then reaches the event vision-based camera 2.

[0051] The first optical path and the second optical path are, for example, an example of the optical path of a laser irradiated with a first phase. In this case, the third optical path is an example of the optical path of a laser irradiated with a second phase.

[0052] The irradiation device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) connected by a bus and a memory 92, and executes a program. The irradiation device 1 functions as a device including the control unit 11, the user interface 12, the light source 13, and the storage unit 14 by executing the program.

[0053] More specifically, the irradiation device 1 causes the processor 91 to read a program stored in the storage unit 14 and store the read program in the memory 92. By the processor 91 executing the program stored in the memory 92, the irradiation device 1 functions as a device including the control unit 11, the user interface 12, the light source 13, and the storage unit 14.

[0054] The control unit 11 controls the operations of various functional units included in the irradiation device 1. The control unit 11 controls, for example, the operation of the light source 13. By the control of the control unit 11, the light source 13 starts laser irradiation. Also, by the control of the control unit 11, the light source 13 ends laser irradiation. By the control of the control unit 11, the light source 13 changes the angle of laser irradiation. By changing the angle of laser irradiation according to time, scanning as shown in, for example, FIG. 2 is performed.

[0055] The user interface 12 is configured to include, for example, input devices such as a mouse, a keyboard, and a touch panel. The user interface 12 may be configured to include an interface for connecting these input devices to the irradiation device 1. Also, the user interface 12 is configured to include, for example, display devices such as a CRT (Cathode Ray Tube) display, a liquid crystal display, and an organic EL (Electro-Luminescence) display. The user interface 12 may be configured to include an interface for connecting these display devices to the three-dimensional measurement device 3.

[0056] The light source 13 is a light source that emits laser light. That is, the light source 13 is a laser device.

[0057] The storage unit 14 is configured by using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information regarding the irradiation device 1.

[0058] The event vision-based camera 2 is a camera that asynchronously detects the luminance change of each pixel and outputs a signal indicating that the luminance change of the pixel has been detected in combination with information indicating the position of the detected pixel and information indicating the detected time. That is, the event vision-based camera 2 is a so-called event vision-based camera.

[0059] Hereinafter, a signal indicating that a luminance change has been detected is referred to as an event detection signal. The luminance change includes a change in which the luminance increases and a change in which the luminance decreases. Therefore, the luminance change has a direction. The direction of the luminance change is, for example, the positive direction when the luminance increases, and the negative direction when the luminance decreases. Therefore, the event detection signal also includes information indicating the direction of the luminance change. The three-dimensional measurement device 3 does not need to use all of the input event detection signals for estimating the three-dimensional shape of the irradiation target 9. For example, it may use an event detection signal in one of the two directions of the luminance change and not use the other event detection signal.

[0060] As described above, the event vision-based camera 2 outputs an event detection signal, information indicating the pixel in which the luminance change of the pixel has been detected (hereinafter referred to as "detected pixel information"), and information indicating the time when the luminance change of the pixel has been detected (hereinafter referred to as "detected time information"). Since the detected pixel information is information indicating the pixel in which the luminance change of the pixel has been detected, it is information indicating the pixel that has output the event detection signal. Since the detected time information is information indicating the time when the luminance change of the pixel has been detected, it is information indicating the time when the event detection signal has been output.

[0061] In the event vision based camera 2, pixels are arranged in an array to form a light receiving section 21. The electromagnetic wave incident on the event vision based camera 2 is, for example, a laser irradiated by the irradiation device 1, the propagation direction of which is changed by the irradiation target 9.

[0062] 3 will be described in more detail. As described above, the event element 210 generates an electric signal by photoelectric conversion, and the generated electric signal is specifically an event detection signal. Since photoelectric conversion is a phenomenon governed by quantum mechanics such as Fermi's Golden Rule, the event detection signal is a signal that is probabilistically output with a probability according to the intensity of the received electromagnetic wave. Each of the event elements 210 is each pixel of the event vision-based camera 2.

[0063] Since the event detection signal is a signal that is output probabilistically with a probability according to the intensity of the received electromagnetic wave, the event detection signal may not be generated within a unit period even if the event element 210 receives light. The phenomenon in which an event detection signal is generated is also called "firing." Hereinafter, the event element 210 that generates the event detection signal is also called the "firing event element 210."

[0064] The relationship between the optical path of the laser and the igniting event element 210 will be described. When estimating the three-dimensional shape of the irradiation target 9, it is desirable to uniquely determine the angle of irradiation for the igniting event element 210. The reason is as follows. If information on the irradiation angle at which a certain event element 210 ignited can be obtained, it is possible to estimate from which angle the laser scattered at a certain position was irradiated, using the positional relationship between the irradiation device 1 and each event element 210. As a result, it is possible to estimate the three-dimensional shape of each layer of the irradiation target 9. For this reason, it is desirable to uniquely determine the angle of irradiation for the igniting event element 210. In other words, it is desirable that the time series of data indicating the presence or absence of ignition for one event element 210 is a time series indicating only one ignition in a unit period. Hereinafter, the time series of data indicating the presence or absence of ignition is called an event time series.

[0065] However, as described above, since the sites scattered by the irradiation target 9 are different, there may be lasers that enter the same event element 210 even at different irradiation angles. As a result, the event time series of one event element 210 may indicate that multiple types of firings with different phases of the laser irradiation causing them occur in one or more unit periods. Further, although the firing is generated by the photoelectric conversion as described above, as is well known, thermal noise (i.e., photons) is also converted into electrons. As a result, firing may not occur even though no light is being received. As a result, without analyzing the event time series, it is impossible to know the cause of the firing, such as whether the firing was caused by the reception of the laser or by noise, or which irradiation phase it belongs to in the case of being caused by the reception of the laser.

[0066] Therefore, the three-dimensional measurement device 3 described later estimates the cause of the firing included in the event time series of each event element 210. However, it is not necessary to estimate the cause for all firings in the estimation. For example, it may be sufficient to estimate only one irradiation phase. By estimating the phases for a plurality of firings with different causes, for example, when the irradiation target 9 has a plurality of layers, three-dimensional measurement can be performed for the plurality of layers. Further, by estimating the phases for a plurality of firings with different causes, for example, when the irradiation target 9 has a plurality of electromagnetic wave scattering sites, the positions of the plurality of electromagnetic wave scattering sites in the three-dimensional space can be estimated.

[0067] The three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on the event detection signal, the detection pixel information, and the detection time information. More specifically, the three-dimensional measurement device 3 also estimates the three-dimensional shape of the irradiation target 9 based on information regarding the phase and angle of the irradiation of the irradiation device 1 (hereinafter referred to as "irradiation setting information") and information indicating the positional relationship between the irradiation device 1, the irradiation target 9, and each pixel of the event vision-based camera 2 (hereinafter referred to as "system positional relationship information").

[0068] Next, the process executed by the three-dimensional measurement device 3, which is the process of estimating the three-dimensional shape of the irradiation target 9, will be described with reference to FIGS. 4 to 6. Based on the event detection signal, the detected pixel information, and the detection time information, the three-dimensional measurement device 3 generates a time series (hereinafter referred to as the "event time series") of data indicating the presence or absence of firing (hereinafter referred to as the "firing presence / absence data") for each pixel. That is, the samples of the event time series are the firing presence / absence data. Hereinafter, the process of generating an event time series for each pixel based on the event detection signal, the detected pixel information, and the detection time information is referred to as the event time series generation process.

[0069] FIG. 4 will be described in more detail. The graph shown in FIG. 4 is an example of a graph of the event time series. The graph of the event time series indicates the timing at which the event detection signal is output from the event vision-based camera 2 (i.e., the timing of firing). Therefore, FIG. 4 more specifically shows the firing timing when the unit period of the period T occurs 6 times.

[0070] FIG. 4 shows that firing occurs multiple times within one unit period. The reason for the multiple firings is, for example, as described in FIG. 1 above, due to the difference in the scattered parts in the irradiation target 9, the scattering of the laser irradiated in the first phase and the scattering of the laser irradiated in the second phase enter one event element 210.

[0071] The three-dimensional measurement device 3 acquires information indicating the temporal change in the strength of the cross-correlation (hereinafter referred to as the "event correlation strength") between the event time series and the template function. Hereinafter, the process of acquiring information indicating the temporal change in the event correlation strength (hereinafter referred to as the "correlation strength time series") based on the event time series and the template function is referred to as the event correlation strength acquisition process. The template function is, for example, a function representing a Gaussian function repeated at a period T.

[0072] A more detailed description will be given with respect to FIG. 5. The template function shown in FIG. 5 indicates that there are unit periods greater than 5 and less than or equal to 6. FIG. 5 shows that one Gaussian function occurs in each unit period. The Gaussian functions in each unit period have the same half-width regardless of the unit period. Also, the difference between the time of the peak of the Gaussian function in each unit period and the start time of each unit period is the same regardless of the unit period. That is, the phase of the peak of the Gaussian function is the same regardless of the unit period.

[0073] A more detailed description will be given with respect to FIG. 6. FIG. 6 is a diagram showing an example of a graph indicating the correlation intensity time series in the embodiment. The function representing the graph of the cross-correlation intensity time series is, for example, a cross-correlation function obtained by convolving a template function and an event time series. The cross-correlation function indicates the similarity and phase difference between two signals. When two signals are similar and their phase difference is τ, the cross-correlation function has a peak of intensity corresponding to the similarity at the position of time τ. When the phase of the template signal is set to 0, the phase difference represents the phase of the event time series. Therefore, the process of obtaining the strength of the cross-correlation with the template function corresponds to the process of obtaining the phase of each output event detection signal.

[0074] Each value of the half-width and peak time of the Gaussian function is a value predetermined according to the amount of noise and the latency. The amount of noise is the amount of noise generated in the event detection signal. The latency is the difference between the actual time when the luminance change of the electromagnetic wave incident on the event vision-based camera 2 occurs and the time recorded in the event detection signal, and this difference fluctuates probabilistically.

[0075] As described above, since the event detection signal is probabilistically output from the event vision-based camera 2 due to various factors, the latency depends on the probability of the event detection signal being output. Also, as described above, the event detection signal depends on quantum mechanics such as Fermi's golden rule and the amount of event detection signals that the event vision-based camera 2 can output per unit time. Therefore, the event detection signal is a signal that is probabilistically output with a probability corresponding to the intensity of the received electromagnetic wave. Since the output of the event detection signal is a probabilistic phenomenon corresponding to the intensity of the received electromagnetic wave in this way, the index of latency can be defined.

[0076] The lower the latency, that is, the longer the time until the event detection signal is output, the more desirable it is to set the position of the peak within each unit period of the Gaussian function to a later time. The reason is as follows. As described above, the process of obtaining the strength of the cross-correlation with the Gaussian function repeated at the period T corresponds to the process of determining the unit period to which each firing presence / absence data belongs. Therefore, it is desirable that the time of the peak of the Gaussian function substantially coincides with the time of the latency.

[0077] Also, as shown by the theory of Fourier transform, the narrower the time width of the spectrum, the wider the frequency width. Therefore, the wider the half-value width of the Gaussian function in FIG. 5, the narrower the frequency width can be made. Narrowing the frequency width corresponds to a process of further removing signals other than the target frequency component. Therefore, the wider the half-value width of the Gaussian function, the more the influence of noise can be reduced. Therefore, it is desirable that the half-value width of the Gaussian function is wider as the amount of noise is larger. In this way, the template function has a band-pass effect.

[0078] So far, the template function has been described as a function representing the repetition of the Gaussian function. However, the template function may be any function as long as it represents a function repeated at the period T, and may be a function representing the repetition of a function other than the Gaussian function.

[0079] The three-dimensional measurement device 3 estimates the phase of the irradiation that caused ignition for each pixel based on the obtained correlation intensity time series. More specifically, based on the information indicating the position of the start time of a unit period on the time axis of the correlation intensity time series (hereinafter referred to as "start time position information"), among the peaks indicated by the correlation intensity time series, a peak that satisfies the above-mentioned intensity condition, which is a predetermined condition regarding the strength of the event correlation intensity among the peaks belonging to any one unit period, is selected, and the phase of the peak is obtained.

[0080] The intensity condition may be any condition as long as it can select only the peaks indicating ignition caused by the reception of the laser irradiated by the irradiation device 1 among the plurality of peaks indicated by the correlation intensity time series. The intensity condition is, for example, the condition that the Nth peak from the top (N is a predetermined integer of 1 or more) in the order of decreasing strength of the event correlation intensity. The intensity condition may be, for example, the condition that the strength of the event correlation intensity is equal to or greater than a predetermined strength. The intensity condition may be, for example, the condition (hereinafter referred to as "extreme value condition") that among the maximum values within a unit period of the cross-correlation function, those that are equal to or greater than a threshold value. For example, when the intensity condition is the extreme value condition, the phase of the peak can be obtained not only for a predetermined number but also for other peaks. As a result, the three-dimensional measurement device 3 can estimate the three-dimensional shape of the irradiation target 9 with higher accuracy than when the number of peaks for which the phase is obtained is predetermined.

[0081] Since the event correlation intensity is a cross-correlation, the larger the absolute value of the value on the vertical axis of the graph of the ignition presence / absence data, the larger the value. For noise, as described above, since the processing corresponding to the band-pass is performed by a template function having a half-value width set in advance considering the influence of noise, the correlation intensity is low. Therefore, the position of the peak of the mutual intensity time series that satisfies the intensity condition is the timing of ignition caused by the reception of the laser irradiated by the irradiation device 1. Therefore, the phase of the peak of the mutual intensity time series that satisfies the intensity condition is the phase at which ignition occurred due to the reception of the laser irradiated by the irradiation device 1. For example, when the intensity condition is that the strength of the event correlation intensity is the strongest, the position of the peak that satisfies the intensity condition indicates the timing of ignition caused by the reception of the laser irradiated by the irradiation device 1.

[0082] Here, the influence of a missing event, which is an event in which ignition does not occur despite receiving the laser, will be described. A missing event is an event with a low occurrence frequency. Therefore, as a result of the action of the template function, the strength of the event correlation intensity of the missing event is reduced by the band-pass function of the template function. Therefore, even if there is a missing event, the three-dimensional measurement device 3 can estimate the timing of ignition caused by the reception of the laser irradiated by the irradiation device 1. Note that since there is a limit to the amount of event detection signals that the event vision-based camera 2 can output per unit time, the event detection signals are output little by little, and as a result, an event in which the event detection signals are output with a latency exceeding the unit period is an example of a missing event.

[0083] In this way, the three-dimensional measurement device 3 acquires, as the phase at which ignition occurred, the phase of the peak that satisfies the intensity condition among the peaks belonging to any one unit period, which is the peak indicated by the correlation intensity time series. Hereinafter, the process of estimating the phase at which ignition occurred based on the correlation intensity time series is referred to as the ignition phase estimation process. As described above, since there is a one-to-one relationship between the phase of irradiation and the angle of irradiation, estimating the phase at which ignition occurred is estimating the angle of irradiation of the laser that caused the ignition.

[0084] The three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on the result of the ignition phase estimation process. Specifically, for example, the three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on the information indicating the phase at which one or more ignitions occurred, acquired for each pixel, the irradiation setting information, and the system positional relationship information. For example, based on the irradiation setting information, it is estimated which laser that irradiated each pixel at each peak caused the ignition, and based on the result of the estimation and the system positional relationship information, the three-dimensional shape of the irradiation target 9 is estimated according to optical theories such as geometric optics. Hereinafter, the process of estimating the three-dimensional shape of the irradiation target 9 based on the result of the ignition phase estimation process, the irradiation setting information, and the system positional relationship information as described above is referred to as the shape estimation process.

[0085] FIG. 7 is a diagram showing an example of the hardware configuration of the three-dimensional measurement device 3 of the embodiment. The three-dimensional measurement device 3 includes a control unit 31 including a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. The three-dimensional measurement device 3 functions as a device including a control unit 31, an input unit 32, a communication unit 33, a storage unit 34, and an output unit 35 by executing a program.

[0086] More specifically, the three-dimensional measurement device 3 reads out the program stored in the storage unit 34 by the processor 93 and stores the read program in the memory 94. By the processor 93 executing the program stored in the memory 94, the three-dimensional measurement device 3 functions as a device including a control unit 31, an input unit 32, a communication unit 33, a storage unit 34, and an output unit 35.

[0087] The control unit 31 controls the operations of various functional units included in the three-dimensional measurement device 3. The control unit 31 executes, for example, an event time series generation process. The control unit 31 executes, for example, an event correlation intensity acquisition process. The control unit 31 executes, for example, an ignition phase estimation process. The control unit 31 executes, for example, a shape estimation process.

[0088] The input unit 32 is configured to include an input device such as a mouse, a keyboard, a touch panel, etc. The input unit 32 may be configured to include an interface for connecting these input devices to the three-dimensional measurement device 3. Information indicating either one or both of the values of the half-value width and the peak phase of the Gaussian function included in the template function, for example, may be input to the input unit 32. The user may change either one or both of the values of the half-value width and the peak phase of the Gaussian function included in the template function via the input unit 32.

[0089] The communication unit 33 is configured to include an interface for connecting the three-dimensional measurement device 3 to an external device. The communication unit 33 communicates with the external device via wire or wirelessly. The external device is, for example, the device that is the source of the start time position information. The device that is the source of the start time position information is, for example, the irradiation device 1. The communication unit 33 acquires the start time position information by communicating with the device that is the source of the start time position information.

[0090] The external device is, for example, the event vision-based camera 2. The communication unit 33 acquires an event detection signal, detection pixel information, and detection time information by communicating with the event vision-based camera 2.

[0091] The external device is, for example, the device that is the source of the irradiation setting information. The communication unit 33 acquires the irradiation setting information by communicating with the device that is the source of the irradiation setting information. The external device is, for example, the device that is the source of the system position relationship information. The communication unit 33 acquires the system position relationship information by communicating with the device that is the source of the system position relationship information. The external device is, for example, the device that is the source of the information indicating the template function (hereinafter referred to as "template information"). The communication unit 33 acquires the template information by communicating with the device that is the source of the template information.

[0092] Note that the irradiation setting information, the system position relationship information, or the template information may have been stored in the storage unit 34 in advance, or may be input via the input unit 32.

[0093] The storage unit 34 is configured using a computer-readable storage medium device such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 34 stores various information regarding the three-dimensional measurement device 3. The storage unit 34 stores, for example, various information resulting from the processing executed by the control unit 31. The storage unit 34 may have stored in advance, for example, as described above, irradiation setting information, system positional relationship information, or template information.

[0094] The output unit 35 includes, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 35 may include an interface for connecting these display devices to the three-dimensional measurement device 3.

[0095] FIG. 8 is a diagram showing an example of the functional configuration of the control unit 31 included in the three-dimensional measurement device 3 of the embodiment. The control unit 31 includes an event information acquisition unit 311, an event time series acquisition unit 312, an event correlation intensity acquisition unit 313, a firing phase estimation unit 314, a shape estimation unit 315, a communication control unit 316, a storage control unit 317, and an output control unit 318.

[0096] The event information acquisition unit 311 acquires event information. The event information includes an event detection signal, detection pixel information indicating the pixel that output the event detection signal, and detection time information indicating the time when the event detection signal was output.

[0097] The event time series acquisition unit 312 acquires the event time series for each pixel. The event time series acquisition unit 312 acquires the event time series, for example, by executing event time series generation processing for each pixel. The event correlation intensity acquisition unit 313 executes event correlation intensity acquisition processing. The firing phase estimation unit 314 executes firing phase estimation processing. The shape estimation unit 315 executes shape estimation processing.

[0098] The communication control unit 316 controls the operation of the communication unit 33. The storage control unit 317 controls the operation of the storage unit 34. The output control unit 318 controls the operation of the output unit 35.

[0099] FIG. 9 is a flowchart showing an example of the processing flow executed by the irradiation device 1 and the event vision-based camera 2 in the three-dimensional measurement system 100 of the embodiment.

[0100] The irradiation device 1 starts laser irradiation (step S101). That is, the irradiation device 1 starts laser irradiation on the irradiation target 9. Next, the event vision-based camera 2 receives the laser scattered by the irradiation target 9 (step S102). The event vision-based camera 2 outputs an event detection signal with a probability according to the received light intensity (step S103). The control unit 11 provided in the irradiation device 1 determines whether a predetermined end condition for irradiation (hereinafter referred to as "irradiation end condition") is satisfied (step S104). If the irradiation end condition is satisfied (step S104: YES), the irradiation ends. On the other hand, if the irradiation end condition is not satisfied (step S104: NO), the irradiation device 1 changes the irradiation angle (step S105). After step S101, since the laser is continuously irradiated, the laser irradiated at the changed angle after the change in the irradiation angle is irradiated on the irradiation target 9. The irradiation end condition is, for example, the condition that a predetermined number of scans are completed. The irradiation end condition may be, for example, the condition that a predetermined period has elapsed.

[0101] FIG. 10 is a flowchart showing an example of the processing flow executed by the three-dimensional measurement device 3 in the three-dimensional measurement system 100 of the embodiment.

[0102] The event information acquisition unit 311 acquires event information for a predetermined period (step S201). The predetermined period may be a predetermined period or a period from when the irradiation device 1 starts laser irradiation until it ends.

[0103] Next, after step S201, the event time series acquisition unit 312 acquires the event time series (step S202). In step S202, the event time series acquisition unit 312 generates the event time series for each pixel of the event vision-based camera 2 based on the event information, for example, by executing the event time series generation process for each pixel. Next, the event correlation intensity acquisition unit 313 executes the event correlation intensity acquisition process for each pixel (step S203). That is, the event correlation intensity acquisition unit 313 acquires the correlation intensity time series for each pixel based on the event time series and the template function. In step S203, the template function is read from the storage unit 34, for example.

[0104] Next, the firing phase estimation unit 314 executes the firing phase estimation process for each pixel (step S204). That is, the firing phase estimation unit 314 estimates the phase at which firing occurs for each pixel based on the correlation intensity time series. Next, the shape estimation unit 315 executes the shape estimation process (step S205). That is, the shape estimation unit 315 estimates the three-dimensional shape of the irradiation target 9 based on the result of the firing phase estimation process, the irradiation setting information, and the system positional relationship information.

[0105] Note that the processes of steps S202 to S205 may be executed in parallel with the processes of steps S101 to S105, or may be executed after the completion of the processes of steps S101 to S105. Step S201 may be executed before the start of step S101. However, in this case, since the event detection signal acquired before the start of step S101 is an event detection signal generated by noise, it is a signal that has no correlation with the shape of the irradiation target 9.

[0106] The three-dimensional measurement device 3 of the embodiment configured as described above uses a template function that is a predetermined function of the period T, and estimates the three-dimensional shape of the irradiation target 9 based on the strength of the cross-correlation between the template function and the event time series that is the time series of data indicating the presence or absence of ignition. By using the template function as described above, the influence of noise and the influence of missing events are suppressed, and it is possible to estimate the phase of the laser irradiation with respect to the ignition caused by the reception of the laser. If the phase of the laser irradiation is estimated with respect to the ignition caused by the reception of the laser, the accuracy of the three-dimensional measurement is improved as described above. Therefore, the three-dimensional measurement device 3 can improve the measurement accuracy in three-dimensional measurement using an event vision-based camera.

[0107] The three-dimensional measurement system 100 of the embodiment configured as described above includes the three-dimensional measurement device 3. Therefore, the three-dimensional measurement system 100 can improve the measurement accuracy in three-dimensional measurement using an event vision-based camera.

[0108] (Modification example) Instead of executing the event correlation intensity acquisition process and the ignition phase estimation process, the three-dimensional measurement device 3 may execute a cluster estimation process. In the cluster estimation process, each ignition indicated by the event time series is clustered by a clustering method such as k-means, and a process of estimating a cluster whose distance between clusters is the period T is executed based on the clustering result. Next, in the cluster estimation process, for the estimated cluster, a process of estimating the phase of each element based on the value of each element of the cluster and the start time position information is executed to estimate the phase of the laser irradiation with respect to the ignition caused by the reception of the laser. As described above, since there is a one-to-one relationship between the phase of the irradiation and the angle of the irradiation, estimating the phase of the laser irradiation means estimating the angle of the laser irradiation. In this way, in the cluster estimation process, the phase of the laser irradiation is estimated with respect to the ignition caused by the reception of the laser.

[0109] Note that the definition of the distance between clusters in the cluster estimation process is the time difference with the highest occurrence frequency in the set of differences in the values in the time axis direction of any two elements belonging to the clusters. In the case of ignition caused by the reception of laser light, since it occurs frequently at a period T, even when the cluster estimation process is executed, similar to the execution of the event correlation intensity acquisition process and the ignition phase estimation process, the phase of the laser irradiation is estimated for the ignition caused by the reception of laser light.

[0110] Note that each ignition indicated by the event time series means a sample indicating the presence of ignition among each sample of the event time series. Hereinafter, the control unit 31 that executes the cluster estimation process instead of the execution of the event correlation intensity acquisition process and the ignition phase estimation process is referred to as the control unit 31a.

[0111] FIG. 11 is a diagram showing an example of the configuration of the control unit 31a in the modification. The control unit 31a is different from the control unit 31 in that it includes a cluster estimation unit 319 instead of the event correlation intensity acquisition unit 313 and the ignition phase estimation unit 314.

[0112] FIG. 12 is a flowchart showing an example of the processing flow executed by the three-dimensional measurement device 3 that executes the cluster estimation process in the modification. For the same processing as in FIG. 10, the description will be omitted by attaching the same reference numerals as in FIG. 10.

[0113] Next, after the processing of step S201 is executed, the processing of step S202 is executed. After the execution of step S202, the cluster estimation process is executed (step S203a). Next to step S203a, the shape estimation unit 315 estimates the three-dimensional shape of the irradiation target 9 based on the result of the cluster estimation process (step S205a). Specifically, for example, the three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on the information indicating the phase at which one or more ignitions occurred for each pixel obtained by the cluster estimation process, the irradiation setting information, and the system positional relationship information.

[0114] Note that the three-dimensional measurement device 3 may already have stored the scanning period T in the storage unit 34 in advance, or for example, the peak position may be obtained by frequency-converting the event time series. The process of obtaining the peak position by frequency-converting the event time series is executed, for example, by the control unit 31 or the control unit 31a. More specifically, the process of obtaining the peak position by frequency-converting the event time series is executed, for example, by the cluster estimation unit 319. More specifically, the process of obtaining the peak position by frequency-converting the event time series is executed, for example, by the ignition phase estimation unit 314.

[0115] Note that the event detection signal may pass through a noise removal filter or the like from when it is output by the event vision-based camera 2 until it is input to the three-dimensional measurement device 3. In such a case, since the three-dimensional measurement device 3 acquires an event detection signal with even less noise, the accuracy of estimating the shape of the irradiation target 9 is further improved.

[0116] The three-dimensional measurement system 100 may include a plurality of event vision-based cameras 2. In such a case, it is also possible to estimate the angle-dependent response function of the irradiation target 9 with respect to electromagnetic waves based on the result of estimating the phase of the laser irradiation for the ignition caused by the reception of the laser by the three-dimensional measurement device 3. This is because, since there are a plurality of event vision-based cameras 2, it is possible to obtain information on reflected light with different reflection directions from the reflected light reflected from the same part. The process of estimating the response function is executed by the control unit 31 or the control unit 31a. More specifically, the process of estimating the response function may be executed, for example, by the shape estimation unit 315.

[0117] Also, when the three-dimensional measurement system 100 includes a plurality of event vision-based cameras 2, a more robust estimation of the shape of the irradiation target 9 is possible than when the three-dimensional measurement system 100 includes one event vision-based camera 2.

[0118] Note that when the three-dimensional measurement system 100 includes a plurality of event vision-based cameras 2, it is not necessarily required to use the template function. When the three-dimensional measurement system 100 includes a plurality of event vision-based cameras 2, instead of the process of obtaining the strength of the cross-correlation between the template function and the event time series, a process of obtaining the strength of the cross-correlation between the event time series generated by each event vision-based camera 2 may be executed.

[0119] Note that the arrangements of the irradiation device 1, the event vision-based camera 2, and the irradiation target 9 do not necessarily have to be the arrangements shown in FIG. 1. The irradiation device 1, the event vision-based camera 2, and the irradiation target 9 may form an angle of 90 degrees or more with the line of sight of the irradiation device 1 and the event vision-based camera 2. In such a case, the event vision-based camera 2 receives backscattering.

[0120] Note that when the refractive index inside the irradiation target 9 is non-uniform, the shape estimation unit 315 may estimate the shape of the irradiation target 9 by numerical analysis using Maxwell's equations or the like, using information indicating the distribution of the refractive index inside the irradiation target 9 (hereinafter referred to as "refractive index distribution information"). The refractive index distribution information may be stored in the storage unit 34 in advance, or may be input by the user via the input unit 32.

[0121] Note that the irradiation device 1 and the three-dimensional measurement device 3 may be implemented using a plurality of information processing devices communicably connected via a network.

[0122] Note that so far, the three-dimensional measurement system 100 has been described by taking the case where three-dimensional measurement is performed by laser irradiation as an example. However, the three-dimensional measurement may be performed by irradiating electromagnetic waves, and it is not necessarily required to be performed by laser irradiation.

[0123] So far, the irradiation device 1 has been described as a device that irradiates electromagnetic waves by changing the irradiation angle. However, the irradiation device 1 may irradiate by changing the position instead of the irradiation angle. For example, the irradiation device 1 may be a liquid crystal or DLP (Digital Lighting Projection) type projector. In such a case, it may be lit so that the condition that pixels or lines light up in order, and different pixels or lines light up at different phases, for each pixel or each line is satisfied. Even with such a change in the irradiation position, the scanning illustrated in FIG. 2 can be performed. Also, the irradiation device 1 may irradiate electromagnetic waves by changing not only the position but also the angle and the position so that the condition that the combination of the angle and the position is different at different phases is satisfied. Even in that case, the scanning illustrated in FIG. 2 can be performed.

[0124] Thus, any device may be used as long as it irradiates electromagnetic waves by changing the irradiation angle or position so that the irradiation device 1 scans the scanning target at a predetermined cycle.

[0125] When the irradiation device 1 is not necessarily limited to the angle and irradiates electromagnetic waves by changing the angle or position, the estimation of the phase is to estimate the irradiation angle or position of the electromagnetic waves. Therefore, the ignition phase estimation unit 314 that estimates the phase estimates the irradiation state of the electromagnetic waves according to the irradiation state of the electromagnetic waves of the irradiation device 1. For example, if the irradiation device 1 is a device that emits electromagnetic waves by changing only the angle, the ignition phase estimation unit 314 is a device that estimates the irradiation angle. If the irradiation device 1 is a device that emits electromagnetic waves by changing the position, the ignition phase estimation unit 314 estimates the irradiation position. If the irradiation device 1 is a device that emits electromagnetic waves by changing the angle and the position, the ignition phase estimation unit 314 estimates the irradiation angle and position. Thus, the ignition phase estimation unit 314 estimates the angle or position at which the light detected by each pixel is irradiated from the irradiation device 1 based on the correlation intensity time series.

[0126] Note that, instead of obtaining the event time series by executing the event time series generation process, the event time series acquisition unit 312 may acquire the event time series of each pixel obtained by executing the event time series generation process by an external device. Note that the ignition phase estimation unit 314 is an example of the irradiation state estimation unit.

[0127] Note that all or part of the functions of the irradiation device 1 and the three-dimensional measurement device 3 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. A computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk incorporated in a computer system. The program may be transmitted via an electric communication line.

[0128] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and designs and the like within the scope not departing from the gist of the present invention are also included.

Explanation of Reference Numerals

[0129] 100... Three-dimensional measurement system, 1... Irradiation device, 2... Event vision-based camera, 3... Three-dimensional measurement device, 11... Control unit, 12... User interface, 13... Light source, 14... Storage unit, 21... Light receiving unit, 210... Event element, 31... Control unit, 32... Input unit, 33... Communication unit, 34... Storage unit, 35... Output unit, 311... Event information acquisition unit, 312... Event time series acquisition unit, 313... Event correlation intensity acquisition unit, 314... Ignition phase estimation unit, 315... Shape estimation unit, 316... Communication control unit, 317... Storage control unit, 318... Output control unit, 319... Cluster estimation unit, 91... Processor, 92... Memory, 93... Processor, 94... Memory, 9... Irradiation target

Claims

1. An irradiation device that irradiates an object to be measured with electromagnetic waves, the irradiation device scanning the electromagnetic waves at a predetermined period, an event vision-based camera that detects scattering of the electromagnetic waves irradiated by the object to be measured, and an event time series that is a time series generated based on the output of the event vision-based camera and indicates the presence or absence of ignition, which is a phenomenon in which an event detection signal indicating that a change in luminance of a pixel included in the event vision-based camera has been detected, is acquired for each pixel included in the event vision-based camera; An event correlation intensity acquisition unit that acquires, for each pixel, a correlation intensity time series indicating a change over time in the strength of the cross-correlation between the event time series and a template function that is a predetermined function of the period; An irradiation situation estimation unit that estimates, based on the correlation intensity time series, the angle or position at which the light detected by each pixel is irradiated from the irradiation device; A shape estimation unit that estimates the three-dimensional shape of the object to be measured based on the estimation result of the irradiation situation estimation unit; A three-dimensional measurement device comprising:

2. The three-dimensional measurement device according to claim 1, wherein the template function is a function representing a Gaussian function repeated at the period.

3. Each value of the full width at half maximum and the peak time of the Gaussian function is a value determined in advance according to the amount of noise generated in the event detection signal and the latency, which is the difference between the actual time when the luminance change of the electromagnetic wave incident on the event vision-based camera occurred and the time recorded in the event detection signal and which fluctuates probabilistically. The three-dimensional measurement device according to claim 2.

4. The three-dimensional measurement device according to claim 3, wherein the full width at half maximum is wider as the amount of noise is larger. The three-dimensional measurement device according to claim 3.

5. An event time series acquisition step of acquiring, for each pixel included in an event vision-based camera, an event time series that is a time series generated based on the output of the event vision-based camera that detects scattering of electromagnetic waves irradiated by an object to be measured by an irradiation device that irradiates the object to be measured with electromagnetic waves and scans the electromagnetic waves at a predetermined period, and that indicates the presence or absence of ignition, which is a phenomenon in which an event detection signal indicating that a change in luminance of a pixel included in the event vision-based camera has been detected; An event correlation intensity acquisition step of acquiring, for each pixel, a correlation intensity time series indicating a temporal change in the strength of the cross-correlation between the event time series and a template function that is a predetermined function of the period; An irradiation situation estimation step of estimating, based on the correlation intensity time series, the angle or position at which the electromagnetic wave detected by each pixel was irradiated from the irradiation device; A shape estimation step of estimating the three-dimensional shape of the measurement target based on the result of the estimation in the irradiation situation estimation step; A three-dimensional measurement method having the above.

6. A program for causing a computer to function as the three-dimensional measurement device according to any one of Claims 1 to 4.

Images