Sensing device

Abstract

To enable generation of higher-accuracy distance data.SOLUTION: A sensing device comprises an image sensor and a processing circuit. The processing circuit acquires pixel value data generated by the image sensor upon receiving reflected light from a scene in a plurality of exposure periods respectively, the pixel value data corresponding to the plurality of exposure periods, generates distance data based upon the pixel value data, and then outputs the distance data and the pixel value data.SELECTED DRAWING: Figure 4A

Description

Technical Field

[0001] The present disclosure relates to a sensing device.

Background Art

[0002] Conventionally, various devices have been proposed for obtaining distance data of an object by irradiating the object with light and detecting reflected light from the object. The distance data of the target scene can be converted into, for example, data of a three-dimensional point cloud and utilized. Point cloud data is typically data in which the distribution of points where objects exist in a scene is represented by three-dimensional coordinates.

[0003] Patent Document 1 discloses a system that scans a space with a light beam and obtains distance information to an object by detecting reflected light from the object with an optical sensor. The system generates and outputs information in which a measurement time is associated with each point of the point cloud data.

[0004] Patent Document 2 discloses an apparatus that measures the distance to a structure existing around a vehicle with a laser scanner and generates three-dimensional point cloud data based on the distance data.

[0005] Patent Document 3 discloses a flash lidar system incorporated in a vehicle that measures the distance to an object using ToF (Time of Flight) technology.

[0006] Patent Document 4 discloses an apparatus that scans a space with a light beam, receives reflected light from an object with an image sensor, and generates distance data.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

[0008] One aspect of the present disclosure provides a novel sensing device that outputs data necessary for distance measurement and a novel information processing apparatus that processes data output from the sensing device. [Means for Solving the Problems]

[0009] A sensing device according to one aspect of the present disclosure includes a light receiving device including at least one light receiving element that performs photoelectric conversion, and a processing circuit that controls the light receiving device. The processing circuit causes the light receiving device to receive reflected light from a scene in each of a plurality of exposure periods, and based on the light receiving data from the light receiving device, generates luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods, the luminance data being used to generate distance data of the scene, and outputs the luminance data and timing data indicating the timing of each of the plurality of exposure periods.

[0010] An information processing apparatus according to another aspect of the present disclosure includes a memory and a processing circuit. The processing circuit acquires from a sensing device luminance data indicating a reflected light amount distribution of reflected light from a scene received in each of a plurality of exposure periods and timing data indicating the timing of each of the plurality of exposure periods, records the luminance data and the timing data in the memory, performs image processing on the luminance data, and generates first distance data based on the luminance data after the image processing and the timing data.

[0011] ​The comprehensive or specific aspects of the present disclosure may be implemented by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or may be implemented by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a volatile recording medium or may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). The apparatus may be composed of one or more apparatuses. When the apparatus is composed of two or more apparatuses, the two or more apparatuses may be arranged in one device or may be separately arranged in two or more separate devices. In this specification and the claims, the "apparatus" may mean not only one apparatus but also a system composed of a plurality of apparatuses.

Advantages of the Invention

[0012] According to one aspect of the present disclosure, based on the data output from the sensing device, the information processing device can generate distance data with higher accuracy.

[0013] Additional benefits and advantageous effects in various aspects included in the present disclosure will become apparent from this specification and the drawings. Each of these benefits and / or advantageous effects may be provided individually by various aspects disclosed in this specification and the drawings or some features in each aspect. It is not necessary for all the features to obtain one or more of these benefits and / or advantageous effects.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 11C

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 15A

Figure 15B

Figure 16A

Figure 16B

Figure 17A

Figure 17B

Figure 18

Figure 19

Figure 20A

Figure 20B

Figure 21A

Figure 21B

Figure 22A

Figure 22B

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29A

Figure 29B

Figure 30

Figure 31

Figure 32

Figure 33A

Figure 33B

Figure 34

Modes for Carrying Out the Invention

[0015] In the present disclosure, all or part of a circuit, unit, device, member or part, or block All or part of the functional blocks in the lock diagram can be executed by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or a large scale integration (LSI). The LSI or IC may be integrated on one chip or may be configured by combining a plurality of chips. For example, functional blocks other than memory elements may be integrated on one chip. Here, although it is called an LSI or IC, the name may change depending on the degree of integration and may be called a system LSI, a very large scale integration (VLSI), or an ultra large scale integration (ULSI). A Field Programmable Gate Array (FPGA) programmed after the manufacture of the LSI, or a reconfigurable logic device capable of reconfiguring the bonding relationship inside the LSI or setting up circuit partitions inside the LSI can also be used for the same purpose.

[0016] Furthermore, all or part of the functions or operations of a circuit, unit, device, member, or part can be executed by software processing. In this case, the software is recorded on a non-transitory recording medium such as one or more ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified by the software are executed by the processor and peripheral devices. The system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and required hardware devices such as an interface.

[0017] <Background> Before describing embodiments of the present disclosure, an example of a distance measurement method that can be used in the embodiments of the present disclosure will be described.

[0018] There are several methods for calculating the distance to an object using a light source and a light receiving device. For example, ToF technologies such as direct ToF method and indirect ToF method are generally used. Among these, the direct ToF method is a method of calculating the distance to an object by directly measuring the time from when the light is emitted until it returns. On the other hand, the indirect ToF method is a method of converting the time from when the light is emitted until it returns into light intensity and measuring it. These distance measurement methods use a light source that emits light pulses and a light receiving device equipped with one or more light receiving elements. Hereinafter, as an example of the distance measurement method, an example of the distance measurement method by the indirect ToF method will be described.

[0019] FIG. 1 and FIG. 2 are diagrams for explaining an example of the distance measurement method by the indirect ToF method. In FIGS. 1 and 2, the rectangular portions represent the periods of light pulse projection, arrival of the reflected light at the light receiving element, and three exposures, respectively. The horizontal axis represents time. FIG. 1 shows an example when a light pulse is reflected from a relatively near object. FIG. 2 shows an example when a light pulse is reflected from a relatively far object. In FIGS. 1 and 2, waveform (a) indicates the timing when the light pulse is emitted from the light source, waveform (b) indicates the period when the reflected light of the light pulse reaches the light receiving element, waveform (c) indicates the first exposure period, waveform (d) indicates the second exposure period, and waveform (e) indicates the third exposure period. Let the time width of the light pulse for distance measurement be T0, and the time from when the light pulse is emitted until it is received, that is, the flight time, be Td.

[0020] In this example, the first exposure period starts simultaneously with the start of light projection and ends simultaneously with the end of light projection. The second exposure period starts simultaneously with the end of light projection and ends when the same time as the time width T0 of the light pulse, that is, the same time as the first exposure period, has elapsed. The third exposure period starts simultaneously with the end of the second exposure period and ends when the same time as the time width T0 of the light pulse, that is, the same time as the first exposure period, has elapsed.

[0021] During the first exposure period, among the reflected light, the light that returns early is photoelectrically converted, and the generated charges are accumulated. Q1 represents the energy of the light that was photoelectrically converted during the first exposure period. This energy Q1 is proportional to the amount of charge accumulated during the first exposure period. During the second exposure period, among the reflected light, the light that arrives after the end of the first exposure period and until the time T0 has elapsed is photoelectrically converted, and the generated charges are accumulated. Q2 represents the energy of the light that was photoelectrically converted during the second exposure period. This energy Q2 is proportional to the amount of charge accumulated during the second exposure period. During the third exposure period, among the reflected light, the light that arrives after the end of the second exposure period and until the time T0 has elapsed is photoelectrically converted, and the generated charges are accumulated. Q3 represents the energy of the light that was photoelectrically converted during the third exposure period. This energy Q3 is proportional to the amount of charge accumulated during the third exposure period.

[0022] Since the length of the first exposure period is equal to the time width T0 of the light pulse, in the example of FIG. 1, the time width of the reflected light received during the second exposure period is equal to the flight time Td. In the example of FIG. 1, since the flight time Td is shorter than the time width T0 of the pulse, all of the reflected light returns to the light receiving element by the end of the second exposure period. Therefore, no reflected light is detected during the third exposure period. The charge accumulated during the third exposure period represents noise due to background light. In contrast, during the first exposure period, in addition to the background light, the charges generated by the reception of the reflected light pulse are accumulated. Similarly, for the second exposure period, in addition to the background light, the charges generated by the reception of the reflected light pulse are accumulated.

[0023] Let the output voltage of the light-receiving element due to the charge accumulated during the first exposure period be V1, the output voltage of the light-receiving element due to the charge accumulated during the second exposure period be V2, and the output voltage of the light-receiving element due to the charge accumulated during the third exposure period be V3. As in the example of FIG. 1, when the reflected light is detected during the first and second exposure periods and not detected during the third exposure period, V1 > V3. In the example of FIG. 1, since the time lengths of the three exposure periods are equal, it is assumed that the background noise does not vary during all exposure periods. In this case, the output voltage V3 of the third exposure period in which the reflected light is not detected can be set as the voltage V of the background noise. BG During the first and second exposure periods, both the charge due to the reflected light and the charge due to the background noise are accumulated. Therefore, the voltage V Q1 due to the charge accumulated by the reception of the reflected light during the first exposure period can be expressed by the following equation (1). V Q1 = V1 - V BG (1)

[0024] Similarly, the voltage V Q2 due to the charge accumulated by the reception of the reflected light during the second exposure period can be expressed by the following equation (2). V Q2 = V2 - V BG (2)

[0025] Since the time length of the first exposure period is equal to the time length of the second exposure period, the ratio of Q1 to Q2 is equal to the ratio of T0 - Td to Td. That is, the ratio of T0 - Td to Td is equal to the ratio of V Q1 to V Q2 . Therefore, Td can be expressed by the following equation (3).

Number

[0026] From equations (1), (2), and (3), Td can be expressed by the following equation (4).

Number

[0027] On the other hand, as shown in the example in FIG. 2, if Td is longer than T0 and the reflected light is If the voltage V1 is not changed and returns during the second and third exposure periods, <V3となる。この場合、第1の露光期間はバックグラウンドノイズによる電荷のみを蓄積する。一方、第2の露光期間および第3の露光期間の一方または両方において、反射光パルスの受光による電荷と、バックグラウンドノイズによる電荷の両方が蓄積される。この場合、Tdは、以下の式(5)で表すことができる。

number

[0028] Using the time of flight Td calculated by equation (4) or (5), the distance D can be calculated by D=c×Td / 2 (c is the speed of light).

[0029] FIG. 3 shows an example of three images generated based on the charge signals accumulated during periods A0, A1, and A2 shown in FIG. 1, and a distance image generated from the data of these images. The light-receiving device in this example is an image sensor equipped with a two-dimensional array of multiple light-receiving elements. During each exposure period, a two-dimensional image is generated based on the charge signals accumulated in each light-receiving element. In the example shown in FIG. 3, reflected light pulses are detected during the first and second exposure periods, and only noise components due to background light are detected during the third exposure period. The distance for each pixel is calculated using the pixel values obtained during each of the first, second, and third exposure periods.

[0030] As described above, when there is noise due to background light other than the reflected light pulse, the pixel value of each pixel includes a noise component. In the above equations (4) and (5), it is assumed that the charge accumulated in each light-receiving element due to noise is equal in all exposure periods. However, in reality, the noise of each pixel varies for each exposure period. Just performing the above calculation for each pixel may not be sufficient to remove the influence of noise.

[0031] Generally, a distance measuring device outputs distance image data or three-dimensional point cloud data as shown in FIG. 3 as a result of distance calculation. The distance image data is represented by a set (x, y, d) of the position x in the horizontal direction, the position y in the vertical direction, and the distance d from the reference position, and represents the distance distribution of the objects existing in the scene. The three-dimensional point cloud data is data representing a plurality of points representing the distribution of objects in the scene in three-dimensional coordinates. The three-dimensional point cloud data can be generated, for example, by being converted from the distance image data.

[0032] As described above, in the indirect ToF method, the distance is calculated by obtaining the ratio of the charges accumulated in the light-receiving elements for each exposure period. Therefore, in a distance measuring device that outputs distance image data or three-dimensional point cloud data, the information on the reflectivity of the object is lost. However, the information on the reflectivity may be useful for the recognition process of the object. For example, in a control device that recognizes an object based on data transmitted from one or more distance measuring devices and controls a vehicle such as an autonomous vehicle based on the recognition result, such reflectivity information may be useful. As an example, when the distance image data or the three-dimensional point cloud data output from the distance measuring device contains a lot of noise, the recognition accuracy may be improved by using the luminance data reflecting the reflectivity of the measurement points.

[0033] Based on the above considerations, the inventors of the present invention came up with the configuration of the embodiment of the present disclosure described below.

[0034] A sensing device according to an embodiment of the present disclosure includes a light source, a light receiving device including at least one light receiving element that performs photoelectric conversion, and a processing circuit that controls the light source and the light receiving device. The processing circuit causes the light source to emit light toward the scene at least once, causes the light receiving device to receive the reflected light by the light in each of a plurality of exposure periods, and generates, based on the light reception data from the light receiving device, luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods, the luminance data being used to generate distance data of the scene, and outputs the luminance data and timing data indicating the timing of each of the plurality of exposure periods.

[0035] According to the above configuration, the processing circuit generates, based on the light reception data from the light receiving device, luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods, the luminance data being used to generate distance data of the scene, and outputs the luminance data and timing data indicating the timing of each of the plurality of exposure periods. As a result, an information processing device that has acquired the luminance data and the timing data can generate higher-quality distance image data or three-dimensional point cloud data based on the luminance data and the timing data.

[0036] The processing circuit may generate the distance data and output the distance data and the luminance data in a switched manner. According to the above configuration, it is possible to switch between a mode of outputting distance data and a mode of outputting luminance data and timing data as necessary. For this reason, for example, flexible control such as outputting luminance data having a large data amount only when necessary becomes possible.

[0037] The processing circuit may output the distance data and the luminance data in a switched manner in response to a request from an external device. The external device may be, for example, an information processing device that generates integrated distance data or point cloud data based on data output from a plurality of sensing devices.

[0038] The processing circuit may switch between and output the distance data and the luminance data according to the state of the received light data. The state of the received light data may include various states such as, for example, the amount of noise included in the received light data or the magnitude of each value included in the received light data.

[0039] The processing circuit may calculate the amount of noise in the received light data for at least one of the plurality of exposure periods, output the luminance data when the amount of noise exceeds a threshold value, and output the distance data when the amount of noise does not exceed the threshold value. Thereby, when it is estimated that the amount of noise in the received light data is large and the reliability of the distance data generated by the processing circuit is low, the luminance data can be output instead of the distance data. The luminance data may be sent to an external information processing device having a higher processing ability than the processing circuit and processed by the information processing device. The information processing device can generate higher-quality distance data or point cloud data based on the luminance data acquired from the sensing device.

[0040] The processing circuit may calculate a reflectance from the received light data for at least one of the plurality of exposure periods, output the distance data when the reflectance exceeds a threshold value, and output the luminance data when the reflectance does not exceed the threshold value. Thereby, control such as outputting the distance data when a high reflectance enables generation of highly reliable distance data and outputting the luminance data otherwise becomes possible.

[0041] The processing circuit may repeat a plurality of frame operations. Each of the plurality of frame operations includes causing the light source to emit the light toward the scene, causing the light receiving device to generate the received light data for each exposure period, and outputting at least one selected from the group consisting of a pair of the luminance data and the timing data and the distance data. Including this, a pair of luminance data and timing data, or distance data can be repeatedly output, for example, at short time intervals.

[0042] The processing circuit may determine, for each frame operation, whether to output the combination of the luminance data and the timing data and the distance data. Thereby, for each frame operation, it is possible to output an appropriate one or both of the combination of the luminance data and the timing data and the distance data.

[0043] When the processing circuit outputs the luminance data or the distance data, it may output the luminance data or the distance data with an identifier indicating which of the luminance data and the distance data is included. Thereby, another device that performs processing based on the luminance data or the distance data can easily determine which of the luminance data and the distance data is included in the acquired data.

[0044] The processing circuit may switch and output the distance data and the luminance data for each of a plurality of regions included in the scene. Thereby, when the reliability of the distance data is low only for some regions in the scene, control such as outputting only the luminance data for the region becomes possible.

[0045] When the processing circuit switches between outputting the combination of the luminance data and the timing data and outputting the distance data, it may output data having a fixed value common to the plurality of frame operations.

[0046] An information processing apparatus according to another embodiment of the present disclosure includes a memory and a processing circuit. The processing circuit acquires luminance data indicating a reflected light amount distribution of reflected light from a scene received during each of a plurality of exposure periods from a sensing device, and timing data indicating the timing of each of the plurality of exposure periods, records the luminance data and the timing data in the memory, performs image processing on the luminance data, and generates first distance data based on the luminance data after the image processing and the timing data.

[0047] According to the above configuration, the processing circuit performs image processing on the luminance data, and generates first distance data based on the luminance data after the image processing and the timing data. The image processing may include, for example, a process of reducing noise in the luminance data.

[0048] The processing circuit may transmit a signal to the sensing device, requesting switching between output of second distance data generated inside the sensing device based on the luminance data and the timing data, and output of the luminance data.

[0049] The processing circuit further acquires identification data indicating which of the second distance data and the luminance data has been output from the sensing device, and may switch the processing for the data output from the sensing device based on the identification data.

[0050] The processing circuit identifies the self-position of the sensing device, and when the self-position of the sensing device satisfies a predetermined condition, the processing circuit may transmit a signal to the sensing device, requesting output of the luminance data.

[0051] The processing circuit determines the amount of noise in the luminance data, and when the amount of noise is greater than a reference value, the processing circuit may transmit a signal to the sensing device, requesting output of the luminance data.

[0052] A computer program according to another embodiment of the present disclosure causes a computer to execute the following operations. · Cause a light source to emit light toward a scene at least once. · Cause a light receiving device to receive reflected light by the light during each of a plurality of exposure periods. · Based on the light reception data from the light receiving device, generate luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods, the luminance data being used to generate distance data of the scene. ·Output the luminance data and timing data indicating the timing of each of the plurality of exposure periods.

[0053] A computer program according to another embodiment of the present disclosure causes a computer to execute the following operations. ·Obtain, from a sensing device, luminance data indicating the distribution of the amount of reflected light of the reflected light from the scene received during each of the plurality of exposure periods, and timing data indicating the timing of each of the plurality of exposure periods. ·Record the luminance data and the timing data in the memory. ·Perform image processing on the luminance data. ·Generate first distance data based on the luminance data after the image processing and the timing data.

[0054] Hereinafter, exemplary embodiments of the present disclosure will be specifically described. Note that each of the embodiments described below shows a comprehensive or specific example. The numerical values, shapes, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components not described in the independent claims indicating the highest-level concept are described as optional components. Also, each figure is a schematic diagram and is not necessarily drawn precisely. Furthermore, in each figure, substantially the same components are denoted by the same reference numerals, and redundant descriptions may be omitted or simplified.

[0055] <Embodiment 1> A system according to Exemplary Embodiment 1 of the present disclosure will be described.

[0056] FIG. 4A is a block diagram showing the physical configuration of the system of the present embodiment. The present system includes a control device 200 and a plurality of distance measurement devices 100. The control device 200 is an information processing device that controls the operation of a vehicle, such as an autonomous vehicle. Each distance measurement device 100 can be a sensing device mounted on the vehicle. Each distance measurement device 100 is connected to the control device 200 by wire or wirelessly. Although the system of the present embodiment includes a plurality of distance measurement devices 100, the number of distance measurement devices 100 may be singular.

[0057] The distance measurement device 100 includes a light source 110, a light receiving device 120, a first processing circuit 130, a recording medium 170, and an input / output interface (IF) 150. The control device 200 includes a second processing circuit 230, a recording medium 270, and an input / output interface 210.

[0058] The light source 110 emits light toward the scene. The light receiving device 120 includes a sensor that detects light emitted from the light source 110 and reflected by an object. The first processing circuit 130 controls the light source 110 and the light receiving device 120 to perform an operation based on the indirect ToF method described above. However, in the present embodiment, the distance measurement device 100 does not perform distance calculation itself, but outputs luminance data of each measurement point that is the basis for distance calculation for each exposure period. Thus, in this specification, a device that does not perform distance calculation itself but generates data necessary for distance calculation is referred to as a "distance measurement device". The distance measurement device 100 outputs, as timing data, time data for specifying the start time and end time of each exposure period in addition to the luminance data of each pixel for each exposure period. These luminance data and time data are sent to the control device 200. The control device 200 uses the luminance data and timing data for each exposure period to calculate the distance of each measurement point by the above operation. The control device 200 can generate data of a distance image or a three-dimensional point cloud based on the calculated distance. The control device 200 can further recognize a specific object in the scene based on the data of the distance image or the three-dimensional point cloud, and control the operation of an operating part of the vehicle, such as an engine, a steering wheel, brakes, or an accelerator, based on the recognition result.

[0059] For each of a plurality of exposure periods, the distance measuring device 100 outputs luminance data of a plurality of consecutive measurement points in a target region that spreads one-dimensionally or two-dimensionally. The light receiving device 120 may include an image sensor capable of acquiring a two-dimensional image. In that case, the distance measuring device 100 outputs luminance data of a plurality of two-dimensionally consecutive measurement points corresponding to a plurality of pixels of the image sensor. On the other hand, when the distance measuring device 100 is a sensing device that detects reflected light while changing the light emission direction one-dimensionally, the distance measuring device 100 outputs luminance data of a plurality of one-dimensionally consecutive measurement points. The distance measuring device 100 in the present embodiment generates luminance data of a plurality of consecutive measurement points in a target region that spreads one-dimensionally or two-dimensionally for each exposure period, and outputs the luminance data together with timing data indicating the timing of each exposure period.

[0060] Next, with reference to FIG. 4B, a more specific configuration example of the present embodiment will be described. FIG. 4B is a block diagram showing a more detailed functional configuration of the distance measuring device 100 and the control device 200. In FIG. 4B, only one of the plurality of distance measuring devices 100 shows a specific configuration. The other distance measuring devices 100 may also have the same configuration. Note that the configurations of the distance measuring devices 100 may be different. For example, some of the distance measuring devices 100 may be configured to output general distance data.

[0061] [Configuration of Distance Measuring Device] The distance measuring device 100 shown in FIG. 4B includes a light source 110, a light receiving device 120, a processing circuit 130, and an input / output interface 150. In the example of FIG. 4B, a clock 160 that outputs time data is provided outside the distance measuring device 100. The clock 160 outputs time data to the plurality of distance measuring devices 100. The clock 160 may be provided inside the distance measuring device 100.

[0062] The light source 110 in this embodiment is an output means for flash light that diffuses laser light over a wide range. The light source 110 includes, for example, a laser light source and a scattering plate, and emits light that spreads over a wide range by scattering the laser light with the scattering plate.

[0063] The light receiving device 120 includes an image sensor 121 and optical components (not shown). The optical components include, for example, one or more lenses, and project light from a range of a certain angle of view onto the light receiving surface of the image sensor 121. The optical components may include other optical elements such as prisms or mirrors. The optical components can be designed such that light diffused from a point of an object in the scene converges to a point on the light receiving surface of the image sensor 121.

[0064] The image sensor 121 is a sensor in which a plurality of light receiving elements 122 are two-dimensionally arranged along the light receiving surface. The image sensor 121 includes a plurality of light receiving elements 122, a plurality of charge accumulation units 124, and a plurality of switches 123. A plurality (for example, three) of charge accumulation units 124 are provided corresponding to each of the plurality of light receiving elements 122. The switch 123 is provided for each light receiving element 122, and switches the connection between the light receiving element 122 and the plurality of charge accumulation units 124 corresponding to the light receiving element 122. Each light receiving element 122 generates charge corresponding to the amount of light received by photoelectric conversion for each exposure period. Each charge accumulation unit 124 accumulates the charge generated by the light receiving element 122 during the corresponding exposure period. The number of charge accumulation units 124 corresponding to each light receiving element 122 is the same as or more than the number of exposure periods required for the distance measurement operation by indirect ToF. The switch 123 switches the connection between the light receiving element 122 and the charge accumulation unit 124 in accordance with an instruction from the processing circuit 130 in response to the switching of the exposure period. In the following description, the combination of one light receiving element 122, the charge accumulation unit 124 corresponding to the one light receiving element 122, and the switch 123 corresponding to the one light receiving element 122 may be referred to as a "pixel".

[0065] The image sensor 121 can be, for example, a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The image sensor 121 may have detection sensitivity not only in the visible wavelength range but also in wavelength ranges such as ultraviolet, near-infrared, mid-infrared, and far-infrared. The image sensor 121 may be a sensor that uses a SPAD (Single Photon Avalanche Diode).

[0066] The image sensor 121 can be provided with, for example, an electronic shutter method that performs exposure of all pixels at once, that is, a mechanism of a global shutter. The electronic shutter may be a rolling shutter method that performs exposure row by row, or an area shutter method that performs exposure only for a partial area according to the irradiation range of the light beam. When the electronic shutter is a global shutter method, two-dimensional information can be acquired at once by controlling the shutter in synchronization with the flash light. On the other hand, in the case of a method that changes the exposure timing for each partial pixel like the rolling shutter method, since pixels with mismatched exposure timing cannot receive light, the amount of information that can be acquired decreases. However, this problem can be solved by performing a process of correcting the deviation of the shutter timing for each pixel. As in the modification example described later, when the light source 110 is a beam scanner that emits a light beam with a small spread, the rolling shutter method may be more suitable than the global shutter method in some cases.

[0067] The light receiving device 120 receives light reflected from an object in a scene. The light receiving device 120 outputs, for each frame, data indicating the charge accumulated for each pixel of the image sensor 121 for each exposure period by the aforementioned indirect ToF. In the present embodiment, in the operation of one frame, for each of a plurality of exposure periods, light projection and exposure are repeated a common number of times so that sufficient charge for distance calculation is accumulated in each exposure period. Then, at the stage when the charge accumulation is completed for all exposure periods, the light receiving device 120 outputs data for all pixels for all exposure periods. The light receiving device 120 outputs the above data, for example, at a rate of 30 frames per second. The data output from the light receiving device 120 is recorded in a recording medium 170 such as a memory.

[0068] The recording medium 170 may include a memory such as a ROM or a RAM, for example. The recording medium 170 records various data generated by the processing circuit 130. The recording medium 170 may further store a computer program executed by the processing circuit 130.

[0069] [[ID=B]]The processing circuit 130 determines the timing of light projection of the flash light by the light source 110 and the timing of exposure of the light receiving device 120, and outputs an exposure control signal and a light projection control signal according to the timing. Further, the processing circuit 130 converts the charge accumulated in the charge accumulation unit 124 for each exposure period at each pixel of the light receiving device 120 into a pixel value for each exposure period, and outputs it as array data of pixel values, that is, image data representing the luminance of each pixel. This image data for each exposure period is sent to the control device 200 via the interface 150.

[0070] The processing circuit 130 is an electronic circuit including a processor such as a CPU, for example. The processing circuit 130 executes the processing in the present embodiment by executing a program stored in the recording medium 170, for example. Note that the recording medium 170 may be included in the processing circuit 130.

[0071] In the dotted frame 300 of FIG. 4B, an example of image data for each exposure period sent from the interface 150 to the control device 200 is schematically shown. In this example, exposure is performed in the three exposure periods A0, A1, and A2 shown in FIGS. 1 and 2. The output data from the distance measurement device 100 in this example includes the same number of image data as the number of exposure periods per frame. In the present embodiment, the output data includes, in addition to the above image data, time data for specifying each exposure period necessary for the control device 200 to perform distance calculation, and data indicating the time length of the light pulse emitted from the light source 110. Details of the output data format will be described later.

[0072] The clock 160 is a circuit that outputs detailed time information necessary for controlling the light source 110. The clock 160 measures time with an accuracy of, for example, nanoseconds or microseconds and outputs the data. The clock 160 can be realized by an integrated circuit such as a real-time clock, for example. The clock 160 may be synchronized with a time server. For synchronization, a protocol such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) may be used, for example. Alternatively, time synchronization may be performed based on the time of the control device 200 using GPS information. Note that the method of time synchronization is not limited to the above and is arbitrary. By time synchronization, the distance measurement device 100 can acquire accurate time data.

[0073] [Configuration of Control Device] Next, the configuration of the control device 200 in the present embodiment will be described. As shown in FIG. 4B, the control device 200 includes an input / output interface 210, a processing circuit 230, and a recording medium 270. The control device 200 controls the operation of the operating unit 240 in the vehicle. The operating unit 240 is a device that performs operations related to autonomous driving, such as an engine, a steering wheel, brakes, or an accelerator, for example.

[0074] The interface 210 acquires output data from a plurality of distance measurement devices 100. The interface 210 also acquires map data distributed from an external server (not shown). The map data can be, for example, data of a landmark map. The interface 210 may be configured to acquire other data that the processing circuit 230 uses for processing. The other data can be, for example, data of a color image, inclination, speed, or acceleration acquired by the distance measurement device 100 or other sensors.

[0075] The processing circuit 230 includes a preprocessing unit 231, a point cloud data generation unit 232, an environment recognition unit 233, and an operation control unit 234. The processing circuit 230 is an electronic circuit including a processor such as a CPU or a GPU. The functions of the preprocessing unit 231, the point cloud data generation unit 232, the environment recognition unit 233, and the operation control unit 234 in the processing circuit 230 can be realized, for example, by the processor of the processing circuit 230 executing a program stored in the recording medium 270. In that case, the processor functions as the preprocessing unit 231, the point cloud data generation unit 232, the environment recognition unit 233, and the operation control unit 234. Each of these functional units may be realized by dedicated hardware. Note that the recording medium 270 may be included in the processing circuit 230.

[0076] The preprocessing unit 231 performs image processing such as noise reduction processing on the output data from the distance measurement device 100 acquired via the interface 210 before performing distance calculation. This image processing is referred to as preprocessing in the following description. The noise reduction processing is performed on the image data for each exposure period. Details of the noise reduction processing will be described later. Note that the preprocessing may include processing such as edge extraction or smoothing in addition to the noise reduction processing. is performed. Details of the noise reduction processing will be described later. Note that the preprocessing may include processing such as edge extraction or smoothing in addition to the noise reduction processing.

[0077] The point cloud data generation unit 232 calculates the aforementioned distance for pixels at the same coordinates of each image using the image for each exposure period processed by the preprocessing unit 231 and the time data for each exposure period of each distance measuring device 100. The point cloud data generation unit 232 calculates the distance for each pixel of the image sensor 121 of each distance measuring device 100, that is, for each position on the xy coordinates, and generates distance image data. After generating the distance image data for each distance measuring device 100, the point cloud data generation unit 232 converts the distance of each pixel of the distance image of each distance measuring device 100 into a point on the three-dimensional coordinates with reference to the control device 200 based on the position and orientation data of each distance measuring device 100.

[0078] The processing circuit 230 extracts objects such as automobiles, people, and bicycles from the point cloud data in the unified coordinates generated by the point cloud data generation unit 232. The processing circuit 230 further matches the point cloud data in the unified coordinates with the map data to recognize the state of its surroundings.

[0079] Based on the position of the object in the three-dimensional space specified by the environment recognition unit 233, the operation control unit 234 determines the operation of the operating unit 240 and transmits a control signal to the operating unit 240.

[0080] The operating unit 240 executes an operation according to the control signal transmitted from the operation control unit 234. For example, it executes operations such as starting, accelerating, decelerating, stopping, and changing direction of the vehicle.

[0081] [Operation of the Distance Measuring Device] Next, the operation of the distance measuring device 100 will be described.

[0082] Figure 5 is a flowchart showing the operation of the distance measuring device 100. The processing circuit 130 of the distance measuring device 100 in the present embodiment executes the operations from step S1120 to S1190 shown in Figure 5. The operations of each step will be described below.

[0083] (Step S1120) The processing circuit 130 first determines whether a signal indicating the end of operation is input from an external device (not shown). If there is an end-of-operation signal in step S1120, the processing circuit 130 ends the operation. If there is no end-of-operation signal in step S1120, the process proceeds to step S1130.

[0084] (Step S1130) The processing circuit 130 outputs a control signal to the light receiving device 120, and the light receiving device 120 releases the electronic shutter according to the control signal output by the processing circuit 130. Thereby, the light detection operation for one frame is started.

[0085] (Step S1140) The processing circuit 130 determines whether all of the predetermined exposure periods required to generate the distance data for one frame have ended. In the present embodiment, during each exposure period, light projection and exposure are repeated a predetermined number of times. Thereby, charges are accumulated in each pixel for all exposure periods. If the charge accumulation for all exposure periods has ended in step S1140, the process proceeds to step S1170. If the charge accumulation for all exposure periods has not ended in step S1140, the process proceeds to step S1150.

[0086] (Step S1150) The processing circuit 130 selects one of the exposure periods in which light projection and exposure have not yet been performed and charge accumulation has not been performed among the predetermined exposure periods, and outputs a switching signal to the switch 123. Thereby, the light receiving element 122 in each pixel and the charge accumulation unit 124 that accumulates charges for the exposure period are connected. Details of the switching timing will be described later.

[0087] (Step S1160) The processing circuit 130 refers to the time data from the clock 160 and generates a signal for controlling the light projection timing of the light source 110 and the exposure timing of the light receiving device 120 according to the predetermined exposure start time and exposure end time for the selected exposure period. The light source 110 emits pulsed flash light of a predetermined time length according to the control signal output by the processing circuit 130. The light receiving device 120 performs exposure according to the start time and end time of the predetermined exposure period with reference to the light projection start point of the light source 110, and accumulates the charges generated by the photoelectric conversion during the exposure period in the charge accumulation unit 124 selected in step S1150. Details of the timing of light projection and exposure will be described later.

[0088] By repeating steps S1140 to S1160, the light receiving operation for one frame is completed.

[0089] (Step S1170) When it is determined in step S1140 that the light projection and light reception for the entire exposure period are completed, the processing circuit 130 closes the shutter in the light receiving device 120.

[0090] (Step S1180) The processing circuit 130 reads out the charges for each exposure period accumulated in each charge accumulation unit 124 of each pixel by the series of operations from step S1130 to step S1160, converts the charges into pixel values, and records them on the recording medium 170. The processing circuit 130 further erases the charges in each charge accumulation unit 124.

[0091] (Step S1190) The processing circuit 130 converts the charge for each exposure period read in step S1180 into a pixel value, and generates luminance image data for each exposure period. Further, in addition to the luminance image data for each exposure period, the processing circuit 130 adds timing data for specifying each exposure period for every predetermined number of frames or at the beginning of each frame. The timing data may include, for example, information on the start time and end time of the exposure period. The processing circuit 130 outputs output data including the luminance image data for each exposure period and the timing data for specifying each exposure period via the interface 150. A specific example of the output data will be described later. After the output, the process returns to step S1120.

[0092] The processing circuit 130 repeats the operations from step S1120 to step S1190. This repeating unit may be referred to as a "frame operation". By repeating a plurality of frame operations, data necessary for distance calculation is output for each frame. The output data is sent to the control device 200.

[0093] In the present embodiment, in step S1190, output data including the timing data for each exposure period is generated in addition to the luminance image data for each exposure period. The output data may include data on the light projection time, that is, the time length of the pulse of the light beam.

[0094] [Examples of light projection and light reception operations] Specific examples of the light projection and exposure operations among the operations of the distance measurement device 100 of the present embodiment will be described.

[0095] FIG. 6 is a time chart showing an example of the light projection and exposure operations by the distance measuring device 100. FIG. 6 represents an example of the light projection and exposure operations for one frame on the time axis. In order from the top, it shows the light projection timing of the light source 110, the exposure timing of the light receiving device 120, the shutter opening period of the light receiving device 120, and the readout timing of the charge accumulation unit 124. The shutter of the light receiving device 120 is opened at the point in time when the light projection and exposure for one frame are started in step S1130, and cannot be closed until the exposure operation is completed. The operation of one frame is started by opening the shutter. In step S1150, one of the plurality of exposure periods is selected, and at each pixel, the charge accumulation unit 124 corresponding to the exposure period and the light receiving element 122 are connected by the switch 123.

[0096] In the present embodiment, in each of the exposure periods A0, A1, and A2, pulsed light projection of the flash light by the light source 110 is repeatedly performed. The time length T0 of the pulse can be, for example, about 90 nanoseconds (nsec). Taking the light projection start time as the origin of time, from the start time to the end time of each exposure period, the charge accumulation unit 124 corresponding to the exposure period and the light receiving element 122 are connected by the switch 123. Inside the dotted frame at the upper part of FIG. 6, an example of the timing of two consecutive light projections and exposures in each exposure period is shown.

[0097] In the exposure period A0, the start and end timings of the exposure are the same as the start and end timings of the light projection. For example, the exposure start time is 0 nsec, and the exposure end time is 90 nsec, which is the same as the time length of the pulse to be projected. The switch is connected simultaneously with the light projection and disconnected simultaneously with the end of the light projection. The light projection by the light source 110, the light reception by the light receiving element 122, and the charge accumulation in the charge accumulation unit 124 are repeated a predetermined number of times. Even when the energy of the reflected light is small in one light projection, it can be made measurable by charge accumulation by multiple light receptions.

[0098] When the light projection and exposure for a predetermined number of times are completed during the exposure period A0, the next exposure period A1 is selected. In the newly selected exposure period A1, the charge storage unit 124 corresponding to the exposure period and the light receiving element 122 are connected by the switch 123. Even if the exposure period changes, the light projection timing of the flash light by the light source 110 does not change. On the other hand, in the newly selected exposure period A1, as shown in the enlarged view of FIG. 6, the exposure starts at the time when a time length T0 (for example, 90 nsec) of the pulse has elapsed from the light projection start time, and the exposure ends at the time when T0 has elapsed after the exposure starts. That is, the switch 123 connects the charge storage unit 124 corresponding to the exposure period and the light receiving element 122 so that the exposure starts simultaneously with the end of the light projection and the exposure is performed for the same time length T0 as the pulse length of the light projection. Similar to the exposure period A0, the charge storage unit 124 holds the charges accumulated by the repeated exposures.

[0099] When the light projection and exposure for a predetermined number of times are completed during the exposure period A1, the light projection and exposure in the exposure period A2 are further performed. In the exposure period A2, as shown in the enlarged view of FIG. 6, the exposure starts at the time when 2T0 (for example, 180 nsec) has elapsed from the light projection start time, and the exposure ends at the time when T0 has elapsed after the exposure starts. That is, the exposure starts at the time when the time length T0 of the light projection has further elapsed after the end of the light projection. The exposure time length in the exposure period A2 is also T0, similar to the exposure periods A0 and A1. Also in the exposure period A2, the charges are accumulated in the charge storage unit 124 by repeating the light projection and exposure for a predetermined number of times.

[0100] When the charge accumulation is completed in all of the exposure periods A0, A1, and A2, the processing circuit 130 closes the shutter of the light receiving device 120 (step S1170). Then, the processing circuit 130 reads out the charges corresponding to the respective exposure periods accumulated in the charge storage unit 124 of each pixel. Based on the read charges, the processing circuit 130 generates and outputs luminance image data for each exposure period.

[0101] [Example of data format] Next, an example of the format of the data output by the distance measuring device 100 of the present embodiment will be described.

[0102] Figures 7A and 7B show an example of the output format of image data for each exposure period output from the interface 150 of the distance measuring device 100. The output data in this example includes data with fixed values common to a plurality of frames and data different for each frame. As data common to a plurality of frames, for example, a fixed value is output once at the beginning of the output data, or once for a predetermined number of frames.

[0103] The fixed values include, for example, data on position, direction, field of view angle, pixel arrangement, exposure period A0, exposure period A1, and exposure period A2. "Position" indicates the position of the image sensor 121 inside the vehicle. The position can be, for example, 3-byte data expressed in three-dimensional coordinates with the center of the vehicle as the origin. "Direction" indicates the direction in which the light-receiving surface of the surface of the image sensor 121 faces. The direction can be, for example, 3-byte data representing the normal vector of the light-receiving surface expressed in three-dimensional coordinates with the center of the vehicle as the origin. "Field of view angle" indicates the field of view angle of the image sensor 121 and can be expressed, for example, in 2 bytes. "Pixel arrangement" indicates the number of pixels in the x-direction and y-direction of the image sensor 121 and can be, for example, 1-byte data. "Exposure period A0", "exposure period A1", and "exposure period A2" indicate the time ranges of the respective exposure periods. These time ranges can be expressed, for example, as 1-byte data describing the elapsed time from the start time of the corresponding light projection in nanosecond units. Note that the number of exposure periods per frame may be other than 3. Also, the number of exposure periods may change during operation. For example, separately, the number of exposure periods may be defined as a fixed value, and the time data for each exposure period may be output.

[0104] The output data for each frame includes, for example, data such as the date, time, luminance image for exposure period A0, luminance image for exposure period A1, and luminance image for exposure period A2. The date is data indicating, for example, the year, month, and day, and can be represented in 1 byte. The time is data indicating, for example, the hour, minute, second, millisecond, and microsecond, and can be represented in 5 bytes. The luminance image for exposure period A0 is a set of pixel values converted from the charge of each pixel accumulated during exposure period A0, and can be represented as 1-byte data for each pixel, for example. Similarly, the luminance image for exposure period A1 and the luminance image for exposure period A2 can also be represented as 1-byte data for each pixel, for example.

[0105] [Operation of the control device] Next, an example of the operation of the control device 200 will be described.

[0106] FIG. 8 is a flowchart showing the operation of the control device 200 in the present embodiment. The processing circuit 230 of the control device 200 executes the operations from step S2120 to S2200 shown in FIG. 8. The operations of each step will be described below.

[0107] (Step S2120) The processing circuit 230 determines whether a signal indicating the end of operation is input from an external device (not shown). If the operation end signal is input, the operation is terminated. If the operation end signal is not input, the process proceeds to step S2130.

[0108] (Step S2130) The processing circuit 230 determines whether there is an input of data from the distance measuring device 100. If there is an input of data from the distance measuring device 100, the process proceeds to step S2140. If there is no input of data from the distance measuring device 100 the process returns to step S2120.

[0109] (Step S2140) The processing circuit 230 performs preprocessing on the data acquired from the distance measuring device 100 to improve the accuracy of distance calculation. The preprocessing includes, for example, noise removal processing. Here, it is assumed that the distance measuring device 100 outputs data in the formats shown in FIGS. 7A and 7B. In this case, the preprocessing is performed on the image data corresponding to the exposure period A0, the image data corresponding to the exposure period A1, and the image data corresponding to the exposure period A2. The processing circuit 230 performs noise removal processing individually on the image data corresponding to each exposure period. As a method of noise removal processing, for example, adaptive filter processing using a Wiener filter can be performed.

[0110] The processing circuit 230 removes noise by removing high-spatial-frequency components in each image data through filter processing. The filter may be a filter other than the Wiener filter, for example, a Gaussian filter. In order to remove high-spatial-frequency components in each image data, for example, smoothing processing by convolution operation using a predefined filter such as a Laplacian filter may be performed. As preprocessing, noise removal processing other than adaptive filter processing may be performed. Also, signal processing other than noise removal processing, such as contrast enhancement or edge extraction, may be performed.

[0111] (Step S2150) The processing circuit 230 calculates the distance for each pixel using the image data for each exposure period on which noise removal processing was performed in step S2140. The processing circuit 230 extracts the pixel values of the same pixel from the image data for each exposure period, calculates the time of flight based on the aforementioned calculation formulas (4) and (5), and further calculates the distance.

[0112] (Step S2160) The processing circuit 230 further calculates the reflectance of the pixels for which the distance has been calculated, using the distance for each pixel calculated in step S2150 and the image data for each exposure period for which noise removal processing has been performed in step S2140. The reflectance is the ratio of the sum value of the pixel values for each exposure period, from which background noise has been removed, to the value at a predefined reflectance of 100% for each distance. The value at a reflectance of 100% can be obtained, for example, by previously measuring for each distance from the light receiving device 120 the pixel value of the reflected light from a reference white board. The value of the reflectance of 100% for each distance can be recorded in advance in the recording medium 270 in the form of a table or the like.

[0113] FIG. 9 shows an example of a table recorded in the recording medium 270. The table in this example defines the correspondence between the distance from the light receiving surface of the light receiving device 120 and the pixel value when detecting the reflected light from a virtual white board with a reflectance of 100% located at that distance. The value of the reflectance of 100% may be recorded as a function of the distance. The table or function that defines the correspondence between the distance and the value of the reflectance of 100% may be transmitted from the distance measuring device 100 to the control device 200.

[0114] (Step S2170) The processing circuit 230 converts the distance data for each pixel calculated in step S2150, that is, the distance image data, into 3D point cloud data. Here, among the pixels of the distance image data, pixels with a distance value of 0 or infinity are not regarded as points of the point cloud data, and the conversion is not performed. The conversion is performed only for the pixels for which a valid distance has been calculated among the pixels of the distance image data. The conversion is performed, for example, as follows. First, referring to the position data of the distance measuring device 100 shown in FIGS. 7A and 7B, in the coordinate system of the distance measuring device 100 Calculate the distance between the origin and the origin of the coordinate system for data integration set in the control device 200. Further, with reference to the data on the direction of the distance measuring device 100 shown in FIGS. 7A and 7B, calculate the amount of rotation of the coordinate axis of the distance measuring device 100 with respect to the coordinate axes of the integrated coordinate system of the control device 200. Based on the calculated distance and the amount of rotation, perform coordinate transformation on the pixels having valid distance values in the distance image generated in step S2150.

[0115] (Step S2180) The processing circuit 230 recognizes the environment around the distance measuring device 100 based on the point cloud data generated in step S2170 and the map data acquired from the outside. The recognition of the environment can be performed by matching the map data and the point cloud data. For example, the processing circuit 230 identifies a fixed object that matches a fixed object included in the map from the point cloud data, and identifies the positional relationship between the fixed object and the distance measuring device 100. Further, the processing circuit 230 groups the point cloud data for each object using the reflectance information for each point calculated in step S2160. Thereby, a moving object such as a person, an animal, a bicycle, or an automobile that is around the distance measuring device 100 and not included in the map can be extracted.

[0116] The matching between the map data and the point cloud data can be performed, for example, as follows. First, refer to the landmark map, select the landmark to be detected, and read information such as the position coordinates of the landmark. Then, based on the output data of the distance measuring device 100, calculate the distance to the landmark. Matching can be performed based on the coordinates of the landmark and the calculated distance to the landmark.

[0117] Since the pre-processed luminance image data and the point cloud data are generated based on the data output from the same distance measuring device 100, there is no deviation in posture. Therefore, the above image data and point cloud data can be superimposed without performing distortion correction. The processing circuit 230 may identify the object by extracting the object overlapping with the object obtained by grouping the point cloud from the luminance image data. Even when it is difficult to clarify the boundary of the object only by grouping the point cloud, by comparing with the result of pixel grouping using the luminance image without positional deviation, the boundary of the object becomes clearer. In this way, by integrating the information obtained from both the luminance image data and the point cloud data, it becomes possible to reproduce the scene more accurately as a three-dimensional space. When extracting an object from the luminance image data, well-known image recognition processing such as object recognition by AI can be used. Further, as pre-processing for the luminance image data, image processing such as edge extraction and contrast enhancement may be performed.

[0118] (Step S2190) Based on the structures such as buildings and the arrangement of the moving body in the three-dimensional space extracted in step S2180, the processing circuit 230 determines the operations of the operating units 240 such as brakes, accelerators, and steering wheels.

[0119] (Step S2200) Based on the operations of the operating unit 240 determined in step S2190, the processing circuit 230 generates and outputs a control signal for controlling the operating unit 240.

[0120] [Effect] As described above, the distance measuring device 100 of the present embodiment generates luminance image data for each exposure period by performing light projection and exposure in each of the plurality of exposure periods. The control device 200 calculates the distance of each pixel based on the luminance image data for each exposure period, and further converts the distance data into three-dimensional point cloud data. The control device 200 performs pre-processing such as noise removal on the luminance image data for each exposure period before distance calculation. As a result, the distance is more accurate It can be calculated. Furthermore, by calculating the reflectance lost due to distance calculation, the accuracy of object extraction or recognition based on point cloud data can be improved.

[0121] The configuration of this embodiment is merely an example, and various modifications are assumed for this embodiment. Hereinafter, some modifications of this embodiment will be described.

[0122] <Modification Example 1 of Embodiment 1> The light source 110 in Embodiment 1 emits flash light that diffuses laser light over a wide range, but the light source 110 may be configured to emit a light beam with a smaller spread than the flash light. By using a light beam such as a laser beam, the energy density of the light can be made higher than when using flash light. Therefore, reflected light from a more distant object can be detected. In order to project light over a wide range, the light source 110 is controlled to change the direction of the light and perform multiple light projections.

[0123] [Configuration of Distance Measuring Device] The configuration of the distance measuring device 100 in this modification example is the same as the configuration shown in FIG. 4B. However, the functions and operations of the light source 110 are different. Hereinafter, the configuration and operation of this modification example will be described centering on the differences from Embodiment 1.

[0124] The light source 110 in this modification example is a beam scanner that can change the emission direction of the light beam. The light source 110 sequentially irradiates a part of the scene with the light beam in response to a command from the processing circuit 130. To realize this function, the light source 110 is provided with a mechanism for changing the emission direction of the light beam.

[0125] FIG. 10 is a diagram showing an example of a light source 110. The light source 110 in this example includes a light-emitting element such as a laser and at least one movable mirror, for example, a MEMS mirror. The light emitted from the light-emitting element is reflected by the movable mirror and directed toward a predetermined region in the scene. The processing circuit 130 can change the emission direction of the light beam by driving the movable mirror. Thereby, for example, the scene can be scanned with the light beam in one dimension or two dimensions.

[0126] A light source capable of changing the emission direction of light by a structure different from the structure having a movable mirror may be used. For example, a light source using a reflective waveguide as disclosed in Patent Document 4 may be used.

[0127] FIG. 11A is a perspective view schematically showing an example of a light source 110 using a reflective waveguide. For reference, the X-axis, Y-axis, and Z-axis orthogonal to each other are schematically shown. The light source 110 includes an optical waveguide array 10A, a phase shifter array 20A, an optical splitter 30, and a substrate 40 on which they are integrated. The optical waveguide array 10A includes a plurality of optical waveguide elements 10 arranged in the Y direction. Each optical waveguide element 10 extends in the X direction. The phase shifter array 20A includes a plurality of phase shifters 20 arranged in the Y direction. Each phase shifter 20 includes an optical waveguide extending in the X direction. The plurality of optical waveguide elements 10 in the optical waveguide array 10A are respectively connected to the plurality of phase shifters 20 in the phase shifter array 20A. An optical splitter 30 is connected to the phase shifter array 20A.

[0128] The light L0 emitted from a light-emitting element (not shown) is input to the plurality of phase shifters 20 in the phase shifter array 20A via the optical splitter 30. The light that has passed through the plurality of phase shifters 20 is input to the plurality of optical waveguide elements 10 in a state where the phase is shifted by a certain amount in the Y direction. The light input to each of the plurality of optical waveguide elements 10 is emitted as a light beam L2 from a light emission surface 10s parallel to the XY plane in a direction intersecting the light emission surface 10s.

[0129] FIG. 11B is a diagram schematically showing an example of the structure of the optical waveguide device 10. The optical waveguide device 10 includes an optical waveguide layer 15 positioned between a first mirror 11 and a second mirror 12 facing each other, and a pair of electrodes 13 and 14 for applying a driving voltage to the optical waveguide layer 15. The optical waveguide layer 15 can be formed of a material whose refractive index changes upon application of a voltage, such as a liquid crystal material or an electro-optic material. The transmittance of the first mirror 11 is higher than that of the second mirror 12. Each of the first mirror 11 and the second mirror 12 can be formed, for example, of a multilayer reflection film in which a plurality of high refractive index layers and a plurality of low refractive index layers are alternately stacked.

[0130] The light input into the optical waveguide layer 15 propagates along the X direction while being reflected by the first mirror 11 and the second mirror 12 within the optical waveguide layer 15. The arrows in FIG. 11B schematically represent the state of light propagation. A part of the light propagating within the optical waveguide layer 15 is emitted to the outside through the first mirror 11.

[0131] By applying a driving voltage to the electrodes 13 and 14, the refractive index of the optical waveguide layer 15 changes, and the direction of the light emitted from the optical waveguide device 10 to the outside changes. In response to the change in the driving voltage, the direction of the light beam L2 emitted from the optical waveguide array 10A changes. Specifically, the emission direction of the light beam L2 shown in FIG. 11A can be changed along a first direction D1 parallel to the X axis.

[0132] FIG. 11C is a diagram schematically showing an example of the phase shifter 20. The phase shifter 20 includes a total reflection waveguide 21 containing a thermo-optical material whose refractive index changes by heat, a heater 22 in thermal contact with the total reflection waveguide 21, and a pair of electrodes 23 and 24 for applying a driving voltage to the heater 22. The refractive index of the total reflection waveguide 21 is higher than the refractive indices of the heater 22, the substrate 40, and the air. Due to the refractive index difference, the light input into the total reflection waveguide 21 propagates along the X direction while being totally reflected within the total reflection waveguide 21.

[0133] By applying a driving voltage to a pair of electrodes 23 and 24, the total reflection waveguide 21 is heated by the heater 22. As a result, the refractive index of the total reflection waveguide 21 changes, and the phase of the light output from the end of the total reflection waveguide 21 shifts. By changing the phase difference between the light outputs from two adjacent phase shifters 20 among the plurality of phase shifters 20 shown in FIG. 11A, the emission direction of the light beam L2 can be changed along the second direction D2 parallel to the Y-axis.

[0134] With the above configuration, the light source 110 can change the emission direction of the light beam L2 two-dimensionally.

[0135] Details such as the operating principle and operating method of the light source 110 as described above are disclosed in, for example, Patent Document 4. The entire disclosure content of Patent Document 4 is incorporated herein by reference.

[0136] In this modification, the processing circuit 130 records, in the recording medium 170 in association with the light beam projection time, the charge accumulated when the light receiving elements in a part of the region in the image sensor 121 receive light for each light beam projection.

[0137] FIG. 12 is a diagram showing an example of data that can be recorded in the recording medium 170. The recording medium 170 stores in association the representative value of the light beam projection time and the time ID representing the representative value of the light beam projection time each time the light beam is projected a specified number of times. Also, the charge accumulated in the charge accumulation unit 124 along with the light beam projection and the time ID corresponding to the light beam projection are stored for each light receiving element 122, that is, for each pixel. Since the light beam projection is performed for each exposure, the charge and the time ID are recorded for each exposure.

[0138] [Operation of the distance measuring device] Next, the operation of the distance measuring device 100 in this modified example will be described. FIG. 13 is a flowchart showing the operation of the distance measuring device 100 in this modified example. In the flowchart shown in FIG. 13, steps S3110 and S3120 are added to the flowchart shown in FIG. 5, and step S1180 is changed to step S3130. Hereinafter, the operations of each step will be described.

[0139] (Step S1120) The processing circuit 130 first determines whether a signal indicating the end of operation is input from an external device (not shown). If there is an operation end signal in step S1120, the processing circuit 130 ends the operation. If there is no operation end signal in step S1120, the process proceeds to step S3110.

[0140] (Step S3110) The processing circuit 130 determines whether the operations for all of one or more light projection directions determined in advance or specified by an external device (not shown) have been completed. If it is determined that the operations have been completed for all the light projection directions, the process proceeds to step S1190. If it is determined that the operations have not been completed for all the light projection directions, the process proceeds to step S3120.

[0141] (Step S3120) The processing circuit 130 selects one of the directions among the one or more light projection directions determined in advance or specified by an external device (not shown) in which light projection has not yet been performed.

[0142] (Step S1130) The processing circuit 130 outputs a control signal to the light receiving device 120, and the light receiving device 120 releases the electronic shutter in accordance with the control signal output by the processing circuit 130. Thereby, the light detection operation for one light projection direction is started.

[0143] (Step S1140) The processing circuit 130 determines whether all of the predetermined exposure periods necessary to generate the distance data have ended. If the charge accumulation in all exposure periods has ended in step S1140, the process proceeds to step S1170. If the charge accumulation in all exposure periods has not ended in step S1140, the process proceeds to step S1150.

[0144] (Step S1150) The processing circuit 130 selects one of the exposure periods that have not yet had light projection and exposure performed and in which charge accumulation has not been performed among the predetermined exposure periods, and outputs a switching signal to the switch 123.

[0145] (Step S1160) The processing circuit 130 refers to the time data from the clock 160 and generates a signal for controlling the light projection timing of the light source 110 and the exposure timing of the light receiving device 120 according to the predetermined exposure start time and exposure end time for the selected exposure period. The light source 110 emits a pulsed light beam having a predetermined time length in the direction determined in step S3120 according to the control signal output by the processing circuit 130. The light receiving device 120 performs exposure according to the start time and end time of the predetermined exposure period based on the light projection start point of the light source 110, and accumulates the charge generated by the photoelectric conversion during the exposure period in the charge accumulation unit 124 selected in step S 1150.

[0146] (Step S1170) If it is determined in step S1140 that the light projection and light reception in all exposure periods have been completed, the processing circuit 130 closes the shutter in the light receiving device 120.

[0147] (Step S3130) The processing circuit 130 reads out the charge in each exposure period accumulated by the series of operations from step S1130 to step S1160, converts the charge of the pixels that have received the reflected light among all the pixels into pixel values, and records the pixel values in the recording medium 170 for each pixel.

[0148] (Step S1190) Based on the data recorded in step S3130, the processing circuit 130 generates luminance image data for each exposure period. Further, the processing circuit 130 generates data on the light beam projection time in addition to the luminance image data for each exposure period. For example, in addition to the luminance image data for each exposure period, the processing circuit 130 adds timing data for identifying each exposure period for every predetermined number of frames or at the beginning of each frame. The timing data may include, for example, information on the start time and end time of the exposure period. The processing circuit 130 outputs output data including the luminance image data for each exposure period, the timing data for identifying each exposure period, and the time data of each pixel via the interface 150. After the output, the process returns to step S1120.

[0149] The processing circuit 130 repeats the operations from step S1120 to step S1190. This repeating unit is referred to as a "frame operation". By repeating a plurality of frame operations, data necessary for distance calculation is output for each frame. The output data is sent to the control device 200.

[0150] [Example of data format] FIGS. 14A and 14B are diagrams showing an example of the output format of the image data for each exposure period output from the distance measuring device 100 in this modification example. In this example, compared with the examples shown in FIGS. 7A and 7B, a fixed value is added with the number of light projection directions (for example, 1 byte). Further, the data for each frame includes the time data (for example, 5 bytes) for each light projection direction of each exposure period. To the data of the luminance value for each exposure period, data of a time ID (for example, 1 byte) for specifying a representative value of the emission time of the light beam used for obtaining the luminance value is added. The correspondence between the time ID and the actual time is recorded in the recording medium 170 as shown in FIG. 12. The data defining the correspondence between the time ID and the actual time is also previously recorded in the recording medium 270 of the control device 200.

[0151] FIGS. 15A and 15B are diagrams showing another example of the output format of image data for each exposure period output from the distance measuring device 100. In this example, a set of pixels that receive the reflected light of the light beam emitted in the same direction is regarded as one block, and data (for example, 4 bytes) for specifying the pixel region of each block is included in the output data for each frame. Each block is, for example, a rectangular pixel region and can be specified by the coordinates of the pixel located at the uppermost left and the coordinates of the pixel located at the lowermost right. The block ID may be included in the block data. In the example of FIG. 15B, after the date data, the data of the pixel region of the block are arranged as many as the number of light projection directions, and then the data of the time for each light projection direction are arranged as many as the product of the number of directions and the number of exposure periods. Subsequently, the image data obtained by converting the charge of each pixel accumulated in the exposure period A0 into a pixel value, the image data corresponding to the charge accumulated in the exposure period A1, and the image data corresponding to the charge accumulated in the exposure period A2 follow. Note that when outputting the block information as in this example, data for specifying the pixel region of the block may be recorded on the recording medium 170. Or data such as luminance values may be recorded on the recording medium 170 for each block group. In the data formats shown in FIGS. 14B, 15A, and 15B, the time at which the luminance data is acquired is given for each pixel or each pixel block. That is, the luminance image data for one frame includes the luminance data at different times. The luminance image data, distance image data, and point cloud data generated from such a data format can be used by being divided into pixels or pixel blocks included in a predetermined time range instead of in frame units. For example, when combining the data acquired from the distance measuring device 100 with the data including the time information acquired from another measuring device or the like, the control device 200 can divide and combine the data into pixels or pixel blocks based on the time information.

[0152] <Modification Example 2 of Embodiment 1> Next, a second modification of Embodiment 1 will be described. In this modification, unlike the above-described examples, the light source 110 is configured to emit a laser light beam with a small spread a plurality of times while changing its direction one-dimensionally. The light receiving device 120, unlike the above-described examples, includes a single or a small number of light receiving elements instead of an image sensor. The orientation of the light receiving device 120 is controlled in accordance with the change in the light projection direction of the light source 110 so that the normal vector of the light receiving surface of the light receiving device 120 coincides with the light projection direction of the light source 110. Therefore, the processing circuit 130 records the data acquired by light reception on the recording medium 170 not for each light receiving element (pixel) but for each light projection direction of the beam. The light projection direction of the beam is specified, for example, by an angle from a reference position.

[0153] The operation of the distance measuring device 100 in this modification is the same as the operation of the distance measuring device 100 in the above-described first modification, and thus the description thereof is omitted. Luminance data corresponding to the charges accumulated by the light receiving device 120 during each exposure period for each light projection is output as a data series for each frame. One frame operation in this modification refers to a set of a series of light projection and exposure operations in a plurality of directions with respect to one axis determined in advance. In one frame operation, for example, the light beam can be sequentially emitted in a total of 720 directions by changing the direction by 0.5 degrees each in a 360-degree range in the horizontal direction.

[0154] FIGS. 16A and 16B are diagrams showing an example of the output format of data output from the distance measuring device 100 in this modification. In this modification, instead of the data of the angular field of view and the pixel arrangement in the examples shown in FIGS. 14A and 14B, the range of the angle of the direction in which the light beam is emitted in one frame operation is output, for example, in 2 bytes as a fixed value. The "direction" in this modification is data expressed as a vector of a three-dimensional coordinate system with the center of the vehicle as the origin and the 0-degree direction set in advance in the distance measuring device 100.

[0155] In this modified example, for the output data for each frame, first, the date (e.g., 1 byte) on which the data of the frame was acquired is output. Subsequently, the luminance value (e.g., 1 byte) and the time (e.g., 5 bytes) corresponding to the charge accumulated during the exposure period A0 are repeatedly output for all the light projection directions. Similarly, for the exposure periods A1 and A2, the luminance value and the time corresponding to the accumulated charge are repeatedly output for all the light projection directions.

[0156] FIGS. 17A and 17B are diagrams showing other examples of the output format of the data output from the distance measurement device 100. In this example, for the output data for each frame, first, the date (e.g., 1 byte) is output, and then the time (e.g., 5 bytes) for each light projection direction in each exposure period is output. Subsequently, the luminance value (e.g., 1 byte) corresponding to the charge accumulated during the exposure period A0 is repeatedly output for all the light projection directions. Similarly, for the exposure periods A1 and A2, the luminance value corresponding to the accumulated charge is repeatedly output for all the light projection directions.

[0157] <Embodiment 2> Next, an exemplary embodiment 2 of the present disclosure will be described.

[0158] In Embodiment 1 and each of its modified examples, the distance measurement device 100 always outputs luminance data indicating the charge accumulated by light reception in each exposure period without calculating the distance. In contrast, in this embodiment, the distance measurement device 100 can calculate the distance for each pixel based on the luminance data and switch between the output of the distance data and the output of the luminance data according to the situation. This switching of the output may be based on an external instruction or a determination based on internal data. After receiving the data from the distance measurement device 100, the control device 200 determines whether it is luminance data or distance data and performs processing according to the type of the data.

[0159] FIG. 18 is a diagram showing the functional configuration of the system in the present embodiment. The hardware configuration in the present embodiment is the same as that shown in FIG. 4B, but the operation is different from that in the first embodiment. Hereinafter, the configuration and operation of the present embodiment will be described centering on the differences from the first embodiment. In the following description, luminance data may be referred to as raw data.

[0160] [Configuration of the distance measuring device] The processing circuit 130 in the present embodiment has a function of calculating the distance for each pixel based on the raw data. The processing circuit 130 switches between the output of distance data and the output of raw data in response to a request from an external control device 200. In the system shown in FIG. 18, the number of distance measuring devices 100 is one, but similar to the example shown in FIG. 4B, the system may include a plurality of distance measuring devices 100.

[0161] The light source 110 in the present embodiment outputs flash light that diffuses laser light over a wide range, similar to the light source 110 in the first embodiment.

[0162] The light receiving device 120 includes an image sensor 121 and optical components such as lenses. The image sensor 121 has a plurality of pixels arranged two-dimensionally. Each pixel includes a light receiving element 122, a plurality of charge accumulation units 124 respectively corresponding to a plurality of exposure periods, and a switch 123 that switches the connection between the light receiving element 122 and each charge accumulation unit 124.

[0163] The processing circuit 130 determines the timing of the projection of the flash light by the light source 110 and the timing of the exposure of the light receiving device 120. Then, according to the determined timing, it sends a light projection control signal to the light source 110 and an exposure control signal to the image sensor 121. The processing circuit 130 further calculates the distance for each pixel based on the signal generated by the charge accumulation unit 124 for each exposure period. The processing circuit 130 outputs distance data indicating the calculated distance for each pixel, that is, distance image data, via the interface 150. On the other hand, when receiving a signal requesting the output of raw data from the control device 200, the processing circuit 130 outputs raw data instead of the distance data.

[0164] [Configuration of the control device] The control device 200 has the same hardware configuration as the control device 200 shown in FIG. 4B. However, in this embodiment, it is different from the first embodiment in that the processing circuit 230 sends a signal requesting the transmission of raw data to the distance measuring device 100 via the interface 210 according to the situation. For example, when more accurate data is required, such as when a lot of noise is included in the distance image data transmitted from the distance measuring device 100, the processing circuit 230 sends a signal requesting raw data to the distance measuring device 100.

[0165] In the configuration of FIG. 18, signals are directly transmitted and received between the interface 150 of the distance measuring device 100 and the interface 210 of the control device 200. Not limited to such a configuration, the communication between the distance measuring device 100 and the control device 200 may be performed via a network such as the Internet. Other devices on the network are located between the distance measuring device 100 and the control device 200 The device may be interposed. The distance measuring device 100 or the control device 200 may communicate with a storage device such as a cloud server or storage via a network. For communication, communication protocols such as http, ftp, TCP or UDP, and IP may be used. A PULL-type communication method may be used, or a PUSH-type communication method may be used. For wired transmission, for example, Ethernet, USB, RS-232C, HDMI (registered trademark), or coaxial cable can be used. Alternatively, for wireless transmission, any wireless communication method such as 3GPP, 3G / 4G / 5G defined by IEEE, wireless LAN, Wi-Fi, Bluetooth (registered trademark), or millimeter wave can be used.

[0166] [Operation of Distance Measuring Device] FIG. 19 is a flowchart showing the operation of the processing circuit 130 of the distance measuring device 100 in the present embodiment. The flowchart shown in FIG. 19 is obtained by replacing step S1190 in the flowchart shown in FIG. 5 with steps S4110 to S4150. The operations from steps S1120 to S1170 are the same as the operations shown in FIG. 5. Hereinafter, the differences from the operations shown in FIG. 5 will be described.

[0167] (Step S4110) When step S1180 is completed, the processing circuit 130 determines whether or not it has received a signal requesting the output of raw data from the control device 200. If the signal has been received, the process proceeds to step S4140. If the signal has not been received, the process proceeds to step S4120.

[0168] (Step S4120) The processing circuit 130 calculates the distance for each pixel based on the value of the charge of each pixel for each exposure period recorded on the recording medium 170. The distance calculation method is the same as the calculation method in step S2150 shown in FIG. 8.

[0169] (Step S4130) The processing circuit 130 converts the distance for each pixel calculated in step S4120 into a pixel value to generate distance image data. The processing circuit 130 generates output data in which an identifier indicating that the data is distance image data is added to the distance image data. A specific example of the output data will be described later.

[0170] (Step S4140) When it is determined in step S4110 that there is a request for output of raw data, the processing circuit 130 converts the value of the charge for each exposure period recorded in the recording medium 170 into a pixel value to generate luminance image data for each exposure period. In this case, the processing circuit 130 generates output data in which timing data indicating the timing of each exposure period is added to the luminance image data for each exposure period. The format of the output data is the same as the format shown in FIGS. 7A and 7B, for example. However, an identifier indicating that the data is raw data, that is, luminance image data, is added to the head of the output data.

[0171] (Step S4150) The processing circuit 130 outputs the output data generated in step S4130 or step S4140 via the interface 150.

[0172] The processing circuit 130 repeatedly executes the operations from step S1120 to step S4150. As a result, in response to a request from the external control device 200, the output of the distance image data and the output of the luminance image data for each exposure period are switched. The switching between the output of the distance image data and the output of the luminance image data for each exposure period can be performed in any frame. Or the processing circuit 130 may switch the output format every one or more predetermined number of frames. In that case, regardless of the request from the control device 200, the processing circuit 130 switches between the output of the distance image data and the output of the luminance image data every predetermined number of frames.

[0173] [Example of data format] Next, an example of the format of the data output by the distance measuring device 100 in the present embodiment will be described.

[0174] FIGS. 20A and 20B are diagrams showing an example of the format of distance image data output from the interface 150 of the distance measuring device 100. In this example, an identifier indicating whether the data format, that is, either distance data or luminance data, is included is added to the head of a fixed value common to a plurality of frames. When the data format is distance data, the fixed value does not include the data of the exposure periods A0 to A2 shown in FIG. 7B. The fixed value can be output, for example, once at the head of the output data or before the head frame at the time of switching of the output data, as in each of the above-described examples.

[0175] The output data for each frame includes the date (e.g., 1 byte) and time (e.g., 5 bytes) when the data of the frame was generated, and data of a value obtained by converting the distance of each pixel into a pixel value (e.g., 1 byte). A distance image is configured from the data of the pixel values of a plurality of pixels.

[0176] FIGS. 21A and 21B are diagrams showing an example of the format of luminance image data output from the distance measuring device 100. The format of the luminance image data in this example is such that an identifier (e.g., 1 byte) indicating that the data format is luminance image data is added to the head of the fixed value in the examples shown in FIGS. 7A and 7B. In other respects, it is the same as the examples shown in FIGS. 7A and 7B.

[0177] FIG. 22A and FIG. 22B are diagrams showing another example of the format of the luminance image data output from the distance measuring device 100. In this example, data indicating the maximum value (for example, 1 byte) of the pixel values measured in advance for each distance is added as a fixed value to the format shown in FIGS. 22A and 22B. This maximum value of the pixel value represents the sum of the pixel values measured in one pixel during the exposure periods A0 to A2 when it is assumed that light is reflected from an object with a reflectance of 100% (for example, a white board) for each of a plurality of preset distances. This maximum value of the pixel value can be recorded such that the step size increases as the distance increases for a plurality of distances within a specific distance range (for example, from 0.1 m to 50 m, etc.). The step size can be determined to be proportional to the logarithm of the distance with a base of 2, for example. The data of the maximum value of the pixel value is measured in advance for each distance and is recorded in a recording medium 270 in a form such as a table as shown in FIG. 9, for example. The data of the maximum value of the pixel value can be used when calculating the reflectance for each pixel by the processing circuit 230 in the control device 200 as described above. Note that such data of the maximum value of the pixel value may also be included in the output data in Embodiment 1 and each modification thereof.

[0178] [Operation of Control Device] FIG. 23 is a flowchart showing an example of the operation of the processing circuit 230 in the control device 200 in the present embodiment. In the operation shown in FIG. 23, step S5110 is added between step S2130 and step S2140 in the flowchart shown in FIG. 8, and steps S5120 to S5140 are added between step S2170 and S2180. Other points are the same as the operation shown in FIG. 8. Hereinafter, the differences from the operation shown in FIG. 8 will be mainly described.

[0179] (Step S5110) When the processing circuit 230 receives data from the distance measuring device 100 in step S2130, it determines whether the data is raw data, that is, luminance image data. This determination is made based on the value of the data format at the fixed value of the input data. If the input data is raw data, the process proceeds to step S2140. If the input data is not raw data, that is, distance image data, the process proceeds to step S2170.

[0180] (Steps S2140~S2160) The processes of steps S2140, S2150, and S2160 are the same as the processes of the corresponding steps in FIG. 8. In step S2140, the processing circuit 230 performs preprocessing such as noise removal processing on the raw data to improve the accuracy of distance calculation. In step S2150, the processing circuit 230 generates distance image data by calculating the distance for each pixel using the image data for each exposure period on which the noise removal processing was performed in step S2140. In step S2160, the processing circuit 230 calculates the reflectance of the pixel for which the distance was calculated using the distance for each pixel calculated in step S2150 and the image data for each exposure period on which the noise removal processing was performed in step S2140. Note that the value of 100% of the reflectance for each distance used in the calculation of the reflectance may be recorded in the recording medium 170 in advance as in the first embodiment, or may be included as a fixed value in the input data as shown in FIGS. 22A and 22B.

[0181] (Step S2170) The processing circuit 230 converts the distance image data generated in step S2150 or the distance image data input from the distance measuring device 100 into point cloud data. This conversion process is the same as the process of step S2170 in FIG. 8.

[0182] (Step S5120) The processing circuit 230 estimates its own position based on the point cloud data generated in step S2170 and the map data acquired from the outside. The own position can be estimated, for example, by performing matching between the map data and the point cloud data. For example, the processing circuit 230 identifies, by matching, a fixed object in the point cloud data that matches a fixed object included in the map. Then, the own position can be estimated based on the value of the distance to the fixed object and the distance to the fixed object obtained from the map.

[0183] (Step S5130) The processing circuit 230 determines whether the position of the control device 200 estimated in step S5120 is in a region that meets a predetermined condition. The condition can be, for example, a condition such as within 10 m around an intersection. If the position of the control device 200 meets the condition, the process proceeds to step S5140. If the position of the control device 200 does not meet the condition, the process proceeds to step S2190.

[0184] (Step S5140) The processing circuit 230 transmits a signal requesting the output of raw data to the distance measuring device 100 via the interface 150. As an example of the case where the estimated position of the control device 200 does not meet the condition in step S5130, there is a case where the reliability of the distance image data input from the distance measuring device 100 is low. In such a case, the processing circuit 230 requests the distance measuring device 100 to output raw data instead of distance data. Thereby, in the next frame, raw data is input from the distance measuring device 100 to the control device 200 instead of distance data.

[0185] The operations of subsequent steps S2180 to S2200 are the same as the corresponding operations in FIG. 8 and are.

[0186] Note that, in the present embodiment, the distance measuring device 100 outputs either distance image data or luminance image data for each exposure period. Instead of such an operation, when the distance measuring device 100 receives a signal requesting luminance image data, it may output both the distance image data and the luminance image data for each exposure period in addition to the distance image data. Further, in the present embodiment, when the control device 200 receives the luminance image data for each exposure period, it always performs preprocessing and distance calculation. Instead of such an operation, the control device 200 may perform preprocessing and distance calculation only when necessary. The control device 200 may record and accumulate the acquired luminance image data for each exposure period in the recording medium 270. Alternatively, the luminance image data for each exposure period may be accumulated by transmitting it to a device such as an external server or storage via a network.

[0187] In the present embodiment, the control device 200 determines whether or not to output a request signal for low data in step S5130 based on its own position. Instead of such an operation, based on the state of the data acquired by the control device 200 from the distance measuring device 100, it may be determined whether or not to output a request signal for low data. For example, when the value of the high-frequency component of the acquired distance image data or the ratio of the high-frequency component to the whole exceeds a predetermined threshold, a request signal for low data may be output. Further, when the data acquired from a plurality of distance measuring devices 100 is inconsistent, a request signal for low data may be output. Alternatively, when the distance image data acquired from the distance measuring device 100 is inconsistent with the map data, such as when a point cloud is located inside a building indicated by the map data, a request signal for low data may be output. For example, when the number of points at positions inconsistent with the map data among the point clouds corresponding to a plurality of pixels in the distance image data acquired from the distance measuring device 100 is equal to or more than a predetermined number, a request signal for low data may be output. Alternatively, when the control device 200 separately acquires image data, when the number of points that cannot be matched between the image data and the distance image data is larger than a predetermined number, a request signal for low data may be output.

[0188] The control device 200 may also acquire measurement results from a measurement device other than the distance measuring device 100 and determine whether to output a raw data request signal based on the measurement results from the other measurement device. The other measurement device may be a gyro sensor. For example, the control device 200 may output a raw data request signal to the distance measuring device 100 when the measurement result of the gyro sensor indicates a rapid change in the time direction, i.e., when a large tremor or impact occurs. Alternatively, the control device 200 may output a raw data request signal to the distance measuring device 100 when the measurement result of the gyro sensor indicates an angle above or below a predetermined threshold, i.e., when the entire system is tilted, such as when traveling on a steep slope. The other measurement device may also be a camera that captures luminance images or videos. For example, the control device 200 may acquire a signal from the camera and output a raw data request signal to the distance measuring device 100 when a pixel with a luminance exceeding a predetermined value is found in a frame of the acquired image or video. Such high-luminance pixels may appear due to strong sunlight or headlight light from an oncoming vehicle. Alternatively, based on the analysis results of the motion vectors obtained from the video, a raw data request signal may be output to the distance measuring device 100. For example, if the motion vector angle is unstable, it is estimated that the system has been subjected to an impact or vibration, and therefore a raw data request signal may be output.

[0189] If the system is a moving body, the control device 200 may determine whether or not to output a raw data request signal based on the operation plan of the moving body. If the operation plan is a specific operation such as acceleration / deceleration, lane change, or backing up, the control device 200 may output a raw data request signal to the distance measuring device 100. In addition, the control device 200 may receive information from sensors in various operating parts of the moving body, such as locking of the brakes and traction. When a signal indicating an abnormal operation such as an abnormality is acquired, a signal requesting raw data may be output to the distance measuring device 100.

[0190] The processing circuit 230 of the control device 200 may determine whether to output a request signal for low data based on the communication state between the distance measuring device 100 and the control device 200. For example, when data exceeding a predetermined data amount is accumulated in a memory (not shown) provided in the interface 210, the processing circuit 230 may output a signal requesting distance image data. As an example, when the data amount received by the interface 210 per second exceeds or is expected to exceed a predetermined data amount, the processing circuit 230 may output a request signal for distance image data.

[0191] [Effect] As described above, according to the present embodiment, the processing circuit 130 of the distance measuring device 100 generates distance data for each pixel in addition to the luminance data of each pixel for each exposure period. Then, the distance data and the luminance data for each exposure period are switched and output in response to a request from an external control device 200. The processing circuit 130 determines for each frame which of a set of luminance data for each exposure period and timing data indicating the timing of each exposure period, and the distance data to output. Further, when the data sent from the distance measuring device 100 does not satisfy a predetermined condition, the processing circuit 230 of the control device 200 requests the distance measuring device 100 to output luminance data.

[0192] By such an operation, a system can be realized in which normally the distance measuring device 100 outputs distance data, and the distance measuring device 100 outputs luminance data for each exposure period only when higher-precision distance measurement is required. The size of the luminance data for each exposure period is larger than the size of the distance data. According to the operation of the present embodiment, the control device 200 can acquire the luminance data of each pixel for each exposure period having a large size only when necessary. Therefore, the communication amount between the control device 200 and the distance measuring device 100 can be reduced.

[0193] By the operation of this embodiment, normally the distance measuring device 100 outputs distance image data. For example, in a complex traffic situation such as around an intersection, the distance measuring device 100 can output luminance data for each exposure period according to an instruction from the control device 200. The control device 200 that has received the luminance data for each exposure period from the distance measuring device 100 can generate more accurate distance information through preprocessing. Also, by calculating the reflectance of each pixel, the recognition performance can be enhanced and the accuracy of environmental recognition can be improved. Furthermore, by directly recording the detailed luminance data for each exposure period or transferring it to another device such as a server via a network, data indicating the state of a complex traffic environment can be retained. Thereby, for example, when an accident occurs, verification data enabling more detailed situation analysis can also be recorded.

[0194] <Modification Example 1 of Embodiment 2> Next, Modification Example 1 of Embodiment 2 will be described. In this modification example, the light source 110 emits a light beam with a smaller spread instead of flash light. The light source 110 is controlled to change the direction of the light and perform multiple light projections in order to project light over a wide range. The configuration of the light source 110 is the same as the configuration in Modification Example 1 of Embodiment 1.

[0195] FIG. 24 is a flowchart showing the operation of the distance measuring device 100 in this modification example. In the operation shown in FIG. 24, steps S3110 and S3120 are added to the operation shown in FIG. 19, and step S1180 is replaced with step S3130. Otherwise, it is the same as the operation shown in FIG. 19. Also, the operation shown in FIG. 24 is such that step S1190 in the operation shown in FIG. 13 is replaced with steps S4110 to S4150. The operations of each step shown in FIG. 24 are the same as the operations of the corresponding steps shown in FIG. 13 or FIG. 19 and thus the description will be omitted.

[0196] By scanning the target scene with a light beam using a light source 110 that can change the emission direction of the light beam as in this modification example, it is possible to obtain data necessary for distance measurement of an object farther away than in the case of using a light source 110 that emits flash light. Also, by the operations from step S4110 to S4140 shown in FIG. 24, in response to a request from an external device, it is possible to switch between output of distance data and output of luminance data for each exposure period.

[0197] Note that, as the configurations of the light source 110 and the light receiving device 120, configurations similar to those of Modification Example 2 of Embodiment 1 may be adopted. That is, a configuration may be adopted in which a light source 110 that emits a light beam and a light receiving device 120 including a single or a small number of light receiving elements are used, and the directions of the light source 110 and the light receiving device 120 are changed to obtain light reception data necessary for distance measurement. Even in such a configuration, the distance data and the luminance data for each exposure period may be switched and output.

[0198] <Modification Example 2 of Embodiment 2> Next, Modification Example 2 of Embodiment 2 will be described. In this modification example, the content of the output data is switched not based on a request from an external device such as the control device 200, but based on the processing result in the processing circuit 130 of the distance measurement device 100. The configuration of this modification example is the same as the configuration of Embodiment 2.

[0199] FIG. 25 is a flowchart showing the operation of the processing circuit 130 in the distance measurement device 100 in this modification example. In the operation shown in FIG. 25, steps S6110 and S6120 are executed instead of step S4110 in the operation shown in FIG. 19. Steps other than steps S6110 and S6120 are the same as the corresponding steps shown in FIG. 19. Hereinafter, differences from the operation shown in FIG. 19 will be described.

[0200] (Step S6110) The processing circuit 130 processes the value of the charge for each pixel in each exposure period read from the charge storage unit 124 as two-dimensional array data according to the pixel array. Random noise can be extracted as a high-frequency component of the spatial frequency in the image. In this modified example, the processing circuit 130 first divides each luminance image of the exposure periods A0, A1, and A2 into a plurality of regions. For example, each luminance image can be divided into eight regions by being divided into two in the y-axis direction and four in the x-axis direction. Here, any of the luminance images of the exposure periods A0, A1, and A2 is divided in the same manner. The processing circuit 130 performs spatial frequency analysis on the two-dimensional data indicating each divided region. As a method of frequency analysis, for example, two-dimensional Fourier transform can be used. The processing circuit 130 extracts the real part of each frequency component obtained as a result of the Fourier transform. The processing circuit 130 calculates the integral of the absolute value for the components having a frequency equal to or higher than a predetermined frequency among the extracted frequency components, and uses this as the noise amount. The processing circuit 130 calculates the above noise amount for each divided region of the three luminance images corresponding to the exposure periods A0, A1, and A2, respectively.

[0201] (Step S6120) The processing circuit 130 evaluates the magnitude of the noise based on the noise amount in each divided region of each luminance image calculated in step S6110. In step S6120, if the noise amount exceeds the threshold value, the process proceeds to step S4140. In step S6120, if the noise amount does not exceed the threshold value, the process proceeds to step S4120. As an example of a method for evaluating the magnitude of the noise, the processing circuit 130 may determine that there is a large amount of noise if the total noise amount of the entire image is larger than a predetermined value in any of the luminance images of the exposure periods A0, A1, and A2. As another example of determining the magnitude of the noise amount, the processing circuit 130 divides If the amount of noise is greater than a predetermined value in a part of the divided regions, it may be determined that there is a lot of noise. For example, if the amount of noise is greater than a predetermined value in one of the eight divided regions, it may be determined that there is a lot of noise. If the amount of noise is greater than a predetermined value in two or more regions, it may be determined that there is a lot of noise. In this way, by dividing each luminance image into a plurality of regions, calculating and determining the amount of noise for each region, even if there is a large amount of noise in a part of the image region, it can be detected without being overlooked.

[0202] In the above method, when it is determined that there is a lot of noise for any of the luminance images of the exposure periods A0, A1, and A2, the processing circuit 130 may determine that "the amount of noise exceeds the threshold value". The method for determining the amount of noise is not limited to the above method and may be other methods.

[0203] In step S6120, if it is determined that the amount of noise exceeds the threshold value, the processing circuit 130 outputs the luminance image data, that is, the raw data, via the interface 150 (S4140). On the other hand, if it is determined that the noise does not exceed the threshold value, the processing circuit 130 generates and outputs distance image data by calculating the distance for each pixel by the method described above (steps S4120 and S4130).

[0204] By repeating the operations from step S1120 to step S4150, the processing circuit 130 can switch between outputting distance image data and outputting luminance image data for each exposure period according to the state of the acquired luminance data. In this modification, as the state of the luminance data, the amount of noise that can be evaluated based on the magnitude of the high-frequency component is used, but it may also be evaluated based on other indicators related to the reliability of the luminance data. For example, a similar determination may be made based on the reflectance for each pixel calculated from the luminance data of each pixel and the data of the maximum value of the luminance value in the case of 100% reflectance for each previously recorded distance. When the index value of the reflectance calculated based on the reflectance of one or more pixels in the three luminance images respectively corresponding to the exposure periods A0, A1, and A2 is lower than a predetermined value, the processing circuit 130 may output the luminance image data for each exposure period instead of the distance image data. Alternatively, when a predetermined proportion of pixels whose luminance values exceed a predetermined threshold exists, the processing circuit 130 may output the luminance image data for each exposure period instead of the distance image data.

[0205] In this modification, in steps S6110 and S6120, the processing circuit 130 determines the necessity of outputting the luminance image data according to the state of the acquired luminance data. Instead of such an operation, when the distance measuring device 100 stops abnormally for some reason, the processing circuit 130 may output the luminance image data for each exposure period. The abnormality of the distance measuring device 100 may be detected by another measuring device such as a gyro sensor.

[0206] To enable the output of the luminance image data when the distance measuring device 100 has stopped abnormally, the recording medium 170 stores the luminance image data of the latest one or more frames together with the time data. When the distance measuring device 100 stops abnormally, the processing circuit 130 reads out the luminance image data for each exposure period accumulated in the recording medium 170 backward from the time point of the abnormal stop and outputs it to the control device 200. As the output destination, not only the control device 200 but also an external system such as a traffic information center may be transmitted via communication.

[0207] In this modification example, a light source 110 that emits flash light is used. However, as in the aforementioned modification example 1, a beam scanner that emits a light beam with a small spread may be used as the light source 110. Further, as in modification example 2 of Embodiment 1, a light source that emits a light beam with a small spread and a light receiving device including a single or a small number of light receiving elements may be used.

[0208] The data formats in the above examples are illustrative, and the distance measuring device 100 may output the same kind of information in other data formats. Further, the distance measuring device 100 may generate luminance data or distance image data in any of the above formats and then reduce the data amount by compressing the data by a predetermined method.

[0209] When a plurality of distance measuring devices 100 transmit data to one control device 200, an identifier for identifying the distance measuring device 100 may be added to the data included in the data formats in the above examples.

[0210] When a plurality of distance measuring devices 100 output raw data to one control device 200, the control device 200 may process the raw data from the plurality of distance measuring devices 100 together. Such processing may include, for example, addition, averaging, or filtering. By processing the raw data output from the plurality of distance measuring devices 100 together, the accuracy of distance calculation can be improved.

[0211] <Modification Example 3 of Embodiment 2> Next, a modification example 3 of Embodiment 2 will be described. In this modification example, the processing circuit 130 of the distance measurement device 100 determines which data, luminance data or distance data, to output for each region of the image instead of for each frame. The processing circuit 130 in this modification example divides the pixel group of the image sensor 121 into a plurality of regions, and for each region, switches and outputs distance data and luminance data for each exposure period. Therefore, the output data for each frame in this modification example includes both raw data and distance data. The output data includes data for specifying each region and data indicating whether each region is output as raw data or distance data. After receiving data from the distance measurement device 100, the processing circuit 230 of the control device 200 similarly divides the image indicated by the data into a plurality of regions, determines whether each region is raw data or distance data, and performs different processes according to the determination result.

[0212] The configuration of the system in this modification example is the same as the configuration shown in FIG. 18. However, the data recorded on the recording medium 270 and the operations of the processing circuits 130 and 230 are different from the example of FIG. 18.

[0213] FIG. 26 is a diagram schematically showing an example of data recorded on the recording medium 270. In this example, when the control device 200 receives image data from the distance measuring device 100, data as shown in FIG. 26 is recorded on the recording medium 270. In the example shown in FIG. 26, the date, the region ID, and the range of the region, and the region ID, the distance, the luminance at the exposure period A0, the luminance at the exposure period A1, and the luminance at the exposure period A3 recorded for each pixel are recorded. For the pixels where the distance data is recorded, the luminance values at the exposure periods A0, A1, and A2 are not recorded and are blank. Conversely, for the pixels where the luminance values at the exposure periods A0, A1, and A2 are recorded, the distance is not recorded and is blank. After the processing circuit 230 acquires the data transmitted from the distance measuring device 100, for the pixels where the distance value is not recorded, the distance is sequentially calculated and recorded from the data of the luminance values at the exposure periods A0, A1, and A2. Thereby, as the process proceeds, the blank distance data is sequentially replaced by the calculated distance data.

[0214] [Operation of Distance Measuring Device] FIG. 27 is a flowchart showing the operation of the distance measuring device 100 in this modified example. The operation shown in FIG. 27 is the same as the operation shown in FIG. 19 except that steps S4110 to S4150 in the operation shown in FIG. 19 are replaced by steps S7110 to S7170. Hereinafter, the differences from the operation shown in FIG. 19 will be mainly described.

[0215] (Step S7110) When the acquisition of the luminance data for each exposure period for one frame is completed by the operations of steps S1120 to S1180, the processing circuit 130 divides each image indicated by the data into a plurality of regions. At this time, each image is divided into a plurality of regions according to the information on the region ID and the region range described in the signal transmitted from the control device 200.

[0216] (Step S7120) The processing circuit 130 determines whether the generation of output data has been completed for all of the regions divided in step S7110. If the generation of output data has been completed for all regions, the process proceeds to step S7140. If there are still regions for which the generation of output data has not been completed, the process proceeds to step S7130.

[0217] (Step S7130) The processing circuit 130 selects one of the regions in which output data has not yet been generated among the regions divided in step S7110.

[0218] (Step S7140) The processing circuit 130 determines, based on the signal transmitted from the control device 200, whether the output of load data is requested as the output data for the region selected in step S7130. If the output of load data is requested for the region, the process returns to step S7120. If the output of load data is not requested for the region, the process proceeds to step S7150.

[0219] (Step S7150) The processing circuit 130 calculates the distance for each pixel in the region selected in step S7130 based on the luminance value for each exposure period by the method described above.

[0220] (Step S7160) The processing circuit 130 records the distance for each pixel calculated in step S7150 in the recording medium 170.

[0221] FIG. 28 is a diagram showing an example of the data recorded in the recording medium 170. A pair of a time ID and a detailed time, and a pair of a region ID and a region range are recorded. Also, for each pixel, the ID of the region containing the pixel, the value of the charge in each exposure period read in step S1180, and the time ID specifying each exposure period are recorded. For the pixels in the region for which the output of load data is not requested, the distance calculated in step S7150 is also recorded. For the pixels in the region for which the output of load data is requested, the distance is not recorded.

[0222] (Step S7170) The processing circuit 130 converts the distance recorded for each pixel into a pixel value to generate distance data. For pixels without distance data, the value of the charge for each exposure period is converted into a luminance value to generate luminance data. The processing circuit 130 outputs the generated image data via the interface 150 together with detailed time data.

[0223] [Example of Data Format] FIGS. 29A and 29B are diagrams showing an example of the format of the output data. In this example, as fixed values common to a plurality of frames, similar to the examples of FIGS. 7A and 7B, the position, direction, angle of view, pixel arrangement, and timing data of each exposure period of the distance measuring device are output.

[0224] In the examples of FIGS. 29A and 29B, the following data are output in order as values that vary for each frame. First, the date and detailed time when the data were acquired and the number of divided regions of the pixel array of the image sensor 12 are output. Subsequently, the start pixel coordinates and end pixel coordinates indicating the range of each region are output for the number of regions. In this modified example, the shape of each region is rectangular, but other shapes such as an ellipse may also be used. Further, the number of regions for which distance data are output, the ID of each region for which distance data are output, the number of regions for which raw data are output, and the ID of each region for which raw data are output follow. Following these pieces of information, the respective distances of the pixels for which raw data output is not requested are output. Further, for each exposure period, the respective luminance values of the pixels for which raw data output is requested are output in order.

[0225] [Operation of the Control Device] FIG. 30 is a flowchart showing an example of the processing executed by the processing circuit 230 of the control device 200 in this modified example. Among the steps shown in FIG. 30, steps S2120, S2130, and S2170 - S2200 are the same as the corresponding steps shown in FIG. 8. The operations of each step will be described below.

[0226] (Step S2120) The processing circuit 230 determines whether a signal indicating the end of operation is input from an external device (not shown). If the end-of-operation signal is input, the operation ends. If the end-of-operation signal is not input, the process proceeds to step S2130.

[0227] (Step S2130) The processing circuit 230 determines whether there is an input of data from the distance measuring device 100. If there is an input of data from the distance measuring device 100, the data is recorded on the recording medium 270, and the process proceeds to step S9110. If there is no input of data from the distance measuring device 100, the process returns to step S2120.

[0228] (Step S9110) The processing circuit 230 acquires data indicating the number of regions when dividing the image indicated by the input data for each frame recorded on the recording medium 270 into a plurality of regions, and the pixel range of each region. Further, the processing circuit 230 determines whether processing has been completed for all regions. If processing has been completed for all regions, the process proceeds to step S9170 and step S2170. If there are unprocessed regions among the plurality of divided regions, the process proceeds to step S9120. Note that in this modification example, the processing from step S9170 to S9190 and the processing from step S2170 to S2200 are performed in parallel, but these may be performed serially.

[0229] (Step S9120) The processing circuit 230 selects one of the regions among the plurality of divided regions for which processing has not yet been performed.

[0230] (Step S9130) The processing circuit 230 refers to the input data for each frame recorded on the recording medium 270, and determines whether the region ID of the region selected in step S9120 is included in the distance region ID. If the region ID of the region is included in the distance region ID, the process returns to step S9110. If the region ID of the region is not included in the distance region ID, that is, if the region ID of the region is included in the Raw region ID, the process proceeds to step S9140.

[0231] (Step S9140) The processing circuit 230 acquires the raw data of the region, that is, the luminance data of each of the exposure periods A0, A1, and A2 shown in FIGS. 29A and 29B, from the input data from the distance measuring device 100 recorded on the recording medium 270. The processing circuit 230 performs preprocessing on the acquired raw - data. As an example of preprocessing, for example, noise removal processing is performed. Noise removal performs noise removal processing using an adaptive filter on each of the luminance images of the exposure period A0, the luminance image of the exposure period A1, and the luminance image of the exposure period A2 in the pixel range extracted as the region. As preprocessing, noise removal processing other than the adaptive filter may be performed. Also. Signal processing other than noise removal processing, such as contrast enhancement or edge extraction, may be performed.

[0232] (Step S9150) The processing circuit 230 performs pixel-by-pixel distance calculation using the image data for each exposure period on which noise removal processing was performed in step S9140 for the pixel range included in the region. From the image data for each exposure period within the region, the pixel values of the same pixel are extracted, and the distance can be obtained based on the above-described calculation formula.

[0233] (Step S9160) The processing circuit 230 stores the distance data for each pixel within the region calculated in step S9150 in the recording medium 270. After executing step S9160, the process returns to step S9110.

[0234] (Step S9170) For the entire area, when distance data is generated, the processing circuit 230 combines the distance of the pixels having distance values in the data input from the distance measuring device 100 with the distance of the pixels whose distances are calculated in step S9150 to generate distance image data. Then, the pixels of this distance image data are clustered according to the distance. As a result, each pixel in the distance image data is classified into one of a plurality of clusters corresponding to a plurality of distance ranges respectively.

[0235] (Step S9180) For each of the clusters generated in step S9170, excluding the clusters of distance 0 and infinite distance, the processing circuit 230 extracts the area around the cluster where the ratio N1 / N2 of the number of pixels N1 within the distance range characterizing the cluster to the number of pixels N2 outside the distance range characterizing the cluster is equal to or greater than a first threshold value and less than a second threshold value greater than the first threshold value. The processing circuit 230 sets the rectangular area closest to the extracted area as the load data request area.

[0236] (Step S9190) The processing circuit 230 divides the area other than the load data request area set in step S9180 into one or more rectangular areas, and sets each rectangular area as a non-load data request area, that is, a distance data request area. For each of the load data request area and the distance data request area, the processing circuit 230 transmits an instruction signal including data indicating the range of the area and a data format specifying the area, that is, a data code indicating whether it is load data or distance data, to the distance measuring device 100.

[0237] FIG. 31 is a diagram showing an example of the format of the instruction signal. In the example of FIG. 31, the instruction signal includes data indicating the number of areas, a data code (that is, a binary value representing load data / distance data), the range of each area, and a data code described in the order of area IDs.

[0238] The processing circuit 230 generates an output signal to the distance measuring device 100 by the operations from step S9170 to step S9190. In parallel, the processing circuit 230 generates a control signal for the autonomous vehicle based on the output of the distance measuring device 100 by the operations from step S2170 to step S2200. The operations from step S2170 to step S2200 are the same as the operations of the corresponding steps shown in FIG. 8. After steps S9190 and S2200, the process returns to step S2120, and the same operations are repeated.

[0239] [Effect] As described above, the processing circuit 230 of the control device 200 in this modification divides the pixel group in the image sensor 121 of the distance measuring device 100 into a plurality of regions based on the distance image data generated based on the output of the distance measuring device 100. Then, for each region, it requests the distance measuring device 100 to output raw data or distance data. In response to the request from the control device 200, the processing circuit 130 of the distance measuring device 100 determines whether to output raw data or distance data for each region. Thereby, for example, for a region where the accuracy of the distance data output from the distance measuring device 100 is low, it requests the output of raw data, and the processing circuit 230 of the control device 200 can perform more detailed signal processing. As a result, the processing circuit 230 can generate high-precision distance data that is difficult to generate in a short time by the processing circuit 130 of the distance measuring device 100 having relatively low performance.

[0240] In this modification, the control device 200 requests the distance measuring device 100 to output raw data or distance data for each region. Instead of such an operation, for example, the distance measuring device 100 may be made to output distance data for all pixels, and for a specific region, in addition to the distance data, an output of raw data may be requested. When requesting an additional output of raw data for a specific region, the instruction signal output from the control device 200 may include, for example, the number of regions for which raw data is to be additionally output and data specifying the respective ranges of those regions.

[0241] FIG. 32 is a flowchart showing an example of the operation of the distance measuring device 100 when an instruction signal for additionally outputting raw data for a specific area is acquired. In the operation shown in FIG. 32, steps S1110 to S1180, S7150, and S7160 are the same as the corresponding steps shown in FIG. 27. In the example shown in FIG. 32, instead of steps S7110, S7120, S7130, and S7240 in the example of FIG. 27, steps S7210 and S7220 are executed. Also, instead of step S7170, step S7230 is executed. These steps will be described below.

[0242] (Step S7210) After the operation of step S1180, the processing circuit 130 records, on the recording medium 170, the area for outputting raw data in addition to the distance data, which is described in the instruction signal transmitted from the control device 200.

[0243] (Step S7220) The processing circuit 130 determines whether distance calculation has been completed for all pixels of the image sensor 121. If the distance calculation for all pixels has been completed, the process proceeds to step S7230. If there are pixels for which the distance calculation has not been completed, the operations of steps S7150 and S7160 are performed, and the process returns to step S7220.

[0244] (Step S7230) The processing circuit 130 outputs the distance data of each pixel recorded on the recording medium 170 and the raw data of each pixel in the area specified by the instruction signal transmitted from the control device 200.

[0245] FIGS. 33A and 33B are diagrams showing an example of the output format in this modification. In this example, following the fixed value, for each frame, the date and time, the time, the distance calculated for all pixels, the number of areas where raw data is output, the respective ranges of the areas where raw data is output, and the luminance indicating the charge accumulated for each of the exposure periods A0, A1, and A2 for the pixels in each area are output. Thus, in addition to the distance information for all pixels, the distance measuring device 100 When additional low data is output for pixels in a specific area, data in the formats shown in FIGS. 33A and 33B can be output.

[0246] FIG. 34 is a diagram showing an example of the operation of the processing circuit 230 when the control device 200 receives data output in the formats shown in FIGS. 33A and 33B. The operation shown in FIG. 34 is the same as the operation shown in FIG. 30, except that steps S9110 to S9160 in the operation shown in FIG. 30 are replaced with steps S9310 to S9340. Hereinafter, the operations different from the example of FIG. 30 will be mainly described.

[0247] (Step S9310) The processing circuit 230 identifies the area where low data is output and its pixel range from the input data for each frame recorded on the recording medium 270, and determines whether the processing for all areas where low data is output has been completed. If the processing for all areas where low data is output has been completed, the process proceeds to steps S9170 and S2170. If there is an unprocessed area among the areas where low data is output, the process proceeds to step S9320.

[0248] (Step S9320) The processing circuit 230 refers to the input data for each frame recorded on the recording medium 270 and selects one of the areas where low data is output and where processing has not yet been performed.

[0249] (Step S9140) The processing circuit 230 obtains the luminance values of the exposure periods A0, A1, and A2 for each pixel within the range of the area selected in step S9320. The processing circuit 230 performs preprocessing on the obtained luminance values of each exposure period. The preprocessing is, for example, the noise removal processing described above.

[0250] (Step S9150) For the pixels included in the region, the processing circuit 230 calculates the distance for each pixel using the image data for each exposure period after performing the noise removal process in step S9140.

[0251] (Step S9330) The processing circuit 230 compares the variance of the distance data for each pixel output from the distance measuring device 100 with the variance of the distance data calculated in step S9150 after performing the noise removal process in step S9140 for the pixel range included in the region. If the variance of the distance data for each pixel calculated in step S9150 is smaller than the variance of the distance data output from the distance measuring device, the process proceeds to step S9340. If the variance of the distance data for each pixel calculated in step S9150 is greater than or equal to the variance of the distance data output from the distance measuring device, the process returns to step S9310.

[0252] (Step S9340) The processing circuit 230 replaces the value of the distance for each pixel of the region recorded on the recording medium 170 with the value of the distance calculated in step S9150. Thereby, the distance data of the region can be corrected to distance data with reduced noise.

[0253] Note that in this modified example, the control device 200 clusters the distance image for each distance range and divides the image into a plurality of regions based on the distribution state of the distance data of the peripheral pixels of the cluster. However, a plurality of regions may be determined by other methods. The range of each region may be fixed without changing it for each frame. Also, regions may be set based on the distribution of the distance data or the magnitude of the noise. Alternatively, the regions may be set according to the speed of the moving body on which the distance measuring device 100 and the control device 200 are mounted. Furthermore, regions may be set by other methods than these.

[0254] When data of different data formats are mixed within one frame as in this modification example, it is difficult to compress the data. Therefore, when compressing output data in which distance data and raw data are mixed within one frame, the distance data and the raw data may be divided and compressed. The control device 200 that receives the data transmitted from the distance measuring device 100 may process the acquired data set in which the distance data and the raw data are mixed and compress it with all pixels unified in the format of the distance data.

[0255] In this modification example, the light source 110 of the distance measuring device 100 emits flash light, but a beam scanner that emits a light beam with a small divergence angle may be used as the light source 110 as in Modification Example 1 of Embodiment 1. When using a beam scanner, the movement of the light projection direction from the light source 110 can be performed in the same manner as in Modification Example 1 of Embodiment 1. Regarding the timing of light projection and light reception and the repetition of exposure associated with the scanning operation, the same operations as in Modification Example 1 of Embodiment 1 can be applied. Regarding the output data, in order to cope with the shift in the distance measurement time associated with performing wide-range distance measurement by scanning with a light beam having a small divergence angle, detailed time data may be included in the output data for each direction of the light beam.

[0256] In the above embodiments, the operation by the indirect ToF method of acquiring received data during three exposure periods has been mainly described, but the present disclosure is not limited to such an operation. For example, the number of exposure periods is not limited to three, and may be two or four or more.

Industrial Applicability

[0257] The technology of the present disclosure can be widely used in devices or systems that perform distance measurement. For example, the technology of the present disclosure can be used as a component of a LiDAR (Light Detection and Ranging) system.

Explanation of Reference Numerals

[0258] 100 Distance measuring device 110 Light source 120 Light receiving device 121 Image sensor 122 Light receiving element 123 Switch 124 Charge storage section 130 Processing circuit 150 Interface 160 Clock 170 Recording medium 200 Vehicle control device 210 Interface 230 Processing circuit 270 Recording medium 240 Actuating section

Claims

1. An image sensor, A processing circuit, Comprising, The processing circuit, Obtains pixel value data corresponding to each of the plurality of exposure periods, generated by the image sensor receiving reflected light from the scene during each of the plurality of exposure periods, Generates distance data based on the pixel value data, Outputs the distance data and the pixel value data, A sensing device.

2. The processing circuit switches and outputs the distance data and the pixel value data in response to a request from an external device, The sensing device according to claim 1.

3. The processing circuit switches and outputs the distance data and the pixel value data according to the state of the pixel value data, The sensing device according to claim 1 or 2.

4. The processing circuit, Calculates the amount of noise in the pixel value data for at least one of the plurality of exposure periods, When the amount of noise exceeds a threshold, outputs the pixel value data, When the amount of noise does not exceed the threshold, outputs the distance data, The sensing device according to any one of claims 1 to 3.

5. The processing circuit, Calculates a reflectance from the pixel value data for at least one of the plurality of exposure periods, When the reflectance exceeds a threshold, outputs the distance data, When the reflectance does not exceed the threshold, outputs the pixel value data, The sensing device according to any one of claims 1 to 3.

6. When the processing circuit outputs the pixel value data or the distance data, it adds an identifier indicating which data of the pixel value data and the distance data is included and outputs the pixel value data or the distance data, the sensing device according to claim 1.

7. The processing circuit switches and outputs the distance data and the pixel value data for each of a plurality of regions included in the scene, the sensing device according to any one of claims 1 to 6.

Images