Sensing device and information processing device

Abstract

To enable generation of distance data of higher accuracy.SOLUTION: A sensing device includes: a light-receiving device provided with at least one light-receiving element for conducting photoelectric conversion; and a processing circuit for controlling the light-receiving device. The processing circuit causes the light-receiving device to receive reflectance from a scene, in a plurality of exposure periods respectively, generates luminance data which indicates a reflectance amount distribution corresponding to the plurality of exposure periods respectively, on the basis of light-reception data from the light-receiving device, and which is used to generate distance data of the scene, and outputs the luminance data, and timing data indicating a timing of the plurality of exposure periods respectively.SELECTED DRAWING: Figure 4A

Description

[Technical field]

[0001] The present disclosure relates to a sensing device and an information processing device. [Background technology]

[0002] Conventionally, various devices have been proposed that obtain distance data of an object by illuminating the object with light and detecting the light reflected from the object. The distance data of a target scene can be converted into, for example, three-dimensional point cloud data and used. Point cloud data is typically data in which the distribution of points at which objects exist in a scene is expressed in three-dimensional coordinates.

[0003] Patent Document 1 discloses a system that acquires distance information to an object by scanning a space with a light beam and detecting reflected light from the object with an optical sensor. The system generates and outputs information in which the measurement time is associated with each point of the point cloud data.

[0004] Patent Document 2 discloses an apparatus that uses a laser scanner to measure the distance to structures present around a vehicle and generates three-dimensional point cloud data based on the distance data.

[0005] Patent Document 3 discloses a flash lidar system that is installed in a vehicle and measures the distance to an object using ToF (Time of Flight) technology.

[0006] Patent Document 4 discloses an apparatus that scans a space with a light beam and receives reflected light from an object with an image sensor to generate distance data. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2009-294128 A [Patent Document 2] JP 2018-185228 A [Patent Document 3] JP 2019-95452 A [Patent Document 4] US Patent Application Publication No. 2018 / 0217258 Summary of the Invention [Problem to be solved by the invention]

[0008] One aspect of the present disclosure provides a novel sensing device that outputs data necessary for distance measurement, and a novel information processing device that processes the data output from the sensing device. [Means for solving the problem]

[0009] A sensing device according to an embodiment of the present disclosure includes a light receiving unit including at least one light receiving element that performs photoelectric conversion, and a processing circuit that controls the light receiving unit. The processing circuit causes the light receiving unit to receive reflected light from a scene during each of a plurality of exposure periods, generates luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods based on the received light data from the light receiving unit, the luminance data being used to generate distance data for the scene, and outputs the luminance data and timing data indicating the timing of each of the plurality of exposure periods.

[0010] According to another aspect of the present disclosure, an information processing device includes a memory and a processing circuit, which acquires, from a sensing device, luminance data indicating a distribution of an amount of reflected light reflected from a scene during each of a plurality of exposure periods and timing data indicating a timing of each of the plurality of exposure periods, records the luminance data and the timing data in the memory, performs image processing on the luminance data, and generates first distance data based on the luminance data after the image processing and the timing data.

[0011] A comprehensive or specific aspect of the present disclosure may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a volatile recording medium or a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). The apparatus may be composed of one or more devices. When the apparatus is composed of two or more devices, the two or more devices may be arranged in one device, or may be arranged separately in two or more separate devices. In this specification and the claims, "apparatus" may mean not only one device, but also a system consisting of multiple devices. Effect of the Invention

[0012] According to one aspect of the present disclosure, an information processing device can generate distance data with higher accuracy based on data output from a sensing device.

[0013] Additional benefits and advantages of the various embodiments contained in the present disclosure will become apparent from the specification and drawings. Each of these benefits and / or advantages may be provided individually by some features of the various embodiments or each embodiment disclosed in the specification and drawings. Not all features are required to obtain one or more of the benefits and / or advantages. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram for explaining an example of a distance measuring method using an indirect ToF method. [Diagram 2] FIG. 1 is a diagram for explaining an example of a distance measuring method using an indirect ToF method. [Diagram 3] FIG. 13 is a diagram showing an example of three images taken for each exposure period and a distance image generated from the data of these images. [Figure 4A]FIG. 2 is a block diagram showing the physical configuration of the system according to the first embodiment. [Figure 4B] FIG. 1 is a block diagram showing the functional configuration of a system according to a first embodiment. [Diagram 5] 4 is a flowchart showing the operation of the distance measuring device. [Figure 6] 4 is a time chart showing an example of a light projection and exposure operation by the distance measuring device. [Figure 7A] FIG. 1 is a first diagram showing an example of an output format of data output from a distance measuring device. [Figure 7B] FIG. 2 is a second diagram showing an example of an output format of data output from a distance measuring device. [Figure 8] 4 is a flowchart showing the operation of the control device in the first embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of a correspondence relationship between a distance and a maximum pixel value. [Figure 10] FIG. 2 is a diagram illustrating an example of a light source. [Figure 11A] FIG. 1 is a perspective view showing a schematic example of a light source using a reflective waveguide. [Figure 11B] 1A and 1B are diagrams illustrating examples of the structure of an optical waveguide element. [Figure 11C] FIG. 1 is a diagram illustrating an example of a phase shifter. [Figure 12] FIG. 2 is a diagram showing an example of data recorded on a recording medium 170. [Figure 13] 13 is a flowchart showing the operation of the distance measuring device according to the first modification of the first embodiment. [Figure 14A] FIG. 11 is a first diagram showing an example of an output format of data output from a distance measuring device in the first modification of the first embodiment. [Figure 14B] FIG. 2 is a second diagram showing an example of an output format of data output from the distance measuring device in the first modification of the first embodiment. [Figure 15A] FIG. 11 is a first diagram showing another example of an output format of data output from the distance measuring device. [Figure 15B] FIG. 2 is a second diagram showing another example of the output format of data output from the distance measuring device. [Figure 16A]FIG. 11 is a first diagram showing yet another example of an output format of data output from the distance measuring device. [Figure 16B] FIG. 2 is a second diagram showing yet another example of the output format of data output from the distance measuring device. [Figure 17A] FIG. 11 is a first diagram showing yet another example of an output format of data output from the distance measuring device. [Figure 17B] FIG. 2 is a second diagram showing yet another example of the output format of data output from the distance measuring device. [Figure 18] FIG. 11 is a diagram illustrating the functional configuration of a system according to a second embodiment. [Figure 19] 10 is a flowchart showing the operation of a processing circuit of the distance measuring device in the second embodiment. [Figure 20A] FIG. 1 is a first diagram showing an example of a format of distance image data output from a distance measuring device. [Figure 20B] FIG. 2 is a second diagram showing an example of a format of distance image data output from a distance measuring device. [Figure 21A] FIG. 1 is a first diagram showing an example of a format of luminance image data output from a distance measuring device. [Figure 21B] FIG. 2 is a second diagram showing an example of a format of luminance image data output from the distance measuring device. [Figure 22A] FIG. 11 is a first diagram showing another example of the format of luminance image data output from the distance measuring device. [Figure 22B] FIG. 2 is a second diagram showing another example of the format of luminance image data output from the distance measuring device. [Diagram 23] 10 is a flowchart showing the operation of a control device in the second embodiment. [Figure 24] 13 is a flowchart showing the operation of the distance measuring device in the first modified example of the second embodiment. [Diagram 25] 13 is a flowchart showing the operation of the distance measuring device in Modification 2 of the second embodiment. [Figure 26] FIG. 13 is a diagram illustrating an example of data recorded on a recording medium in a third modification of the second embodiment. [Figure 27]13 is a flowchart showing the operation of the distance measuring device in the third modification of the second embodiment. [Figure 28] FIG. 13 is a diagram showing an example of data recorded on a recording medium in a third modification of the second embodiment. [Figure 29A] FIG. 13 is a diagram showing an example of a format of output data in a third modified example of the second embodiment. [Figure 29B] FIG. 13 is a diagram showing an example of a format of output data in a third modified example of the second embodiment. [Diagram 30] 13 is a flowchart showing an example of processing executed by a control device in a third modified example of the second embodiment. [Diagram 31] FIG. 11 is a diagram illustrating an example of a format of an instruction signal. [Diagram 32] 10 is a flowchart showing another example of the operation of the distance measuring device. [Figure 33A] FIG. 1 is a first diagram showing an example of an output format. [Figure 33B] FIG. 2 is a second diagram showing an example of an output format. [Diagram 34] 10 is a flowchart showing another example of the operation of the control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In the present disclosure, all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram may be implemented by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Here, although it is called an LSI or an IC, the name may change depending on the degree of integration, and it may be called a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the manufacture of the LSI, or a reconfigurable logic device, which can reconfigure the junction relationship inside the LSI or set up the circuit partition inside the LSI, can also be used for the same purpose.

[0016] Furthermore, all or part of the functions or operations of a circuit, unit, device, member, or section can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0017] <Background> Before describing the embodiments of the present disclosure, an example of a distance measurement method that can be used in the embodiments of the present disclosure will be described.

[0018] There are several distance measurement methods that use a light source and a light receiving device to calculate the distance to an object. For example, ToF technologies such as the direct ToF method and the indirect ToF method are commonly used. Of these, the direct ToF method is a method that calculates the distance to an object by directly measuring the time from when light is emitted until it returns. On the other hand, the indirect ToF method is a method that converts the time from when light is emitted until it returns into light intensity and measures it. These distance measurement methods use a light source that emits light pulses and a light receiving device that has one or more light receiving elements. Below, an example of a distance measurement method using the indirect ToF method will be described as an example of a distance measurement method.

[0019] 1 and 2 are diagrams for explaining an example of a distance measurement method using the indirect ToF method. In FIG. 1 and FIG. 2, the rectangular parts represent the periods of the projection of the light pulse, the arrival of the reflected light at the light receiving element, and the three exposures. The horizontal axis represents time. FIG. 1 shows an example in which the light pulse is reflected from a relatively close object. FIG. 2 shows an example in which the light pulse is reflected from a relatively distant object. In FIG. 1 and FIG. 2, the waveform (a) shows the timing when the light pulse is emitted from the light source, the waveform (b) shows the period in which the reflected light of the light pulse reaches the light receiving element, the waveform (c) shows the first exposure period, the waveform (d) shows the second exposure period, and the waveform (e) shows the third exposure period. The time width of the light pulse for distance measurement is T0, and the time from when the light pulse is emitted to when it is received, that is, the flight time, is Td.

[0020] In this example, the first exposure period starts at the same time as the start of light projection and ends at the same time as the end of light projection. The second exposure period starts at the same time as the end of light projection and ends when the same time as the time width T0 of the light pulse, i.e., the same time as the first exposure period, has elapsed. The third exposure period starts at the same time as the end of the second exposure period and ends when the same time as the time width T0 of the light pulse, i.e., the same time as the first exposure period, has elapsed.

[0021] In the first exposure period, the reflected light that returns early is photoelectrically converted, and the resulting charge is accumulated. Q1 represents the energy of the light photoelectrically converted during the first exposure period. This energy Q1 is proportional to the amount of charge accumulated during the first exposure period. In the second exposure period, the reflected light that arrives after the first exposure period ends and before the time T0 has elapsed is photoelectrically converted, and the resulting charge is accumulated. Q2 represents the energy of the light photoelectrically converted during the second exposure period. This energy Q2 is proportional to the amount of charge accumulated during the second exposure period. In the third exposure period, the reflected light that arrives after the second exposure period ends and before the time T0 has elapsed is photoelectrically converted, and the resulting charge is accumulated. Q3 represents the energy of the light photoelectrically converted during the third exposure period. This energy Q3 is proportional to the amount of charge accumulated during the third exposure period.

[0022] Since the length of the first exposure period is equal to the time width T0 of the light pulse, in the example of FIG. 1, the time width of the reflected light received in the second exposure period is equal to the flight time Td. In the example of FIG. 1, since the flight time Td is shorter than the time width T0 of the pulse, all of the reflected light returns to the light receiving element by the end of the second exposure period. Therefore, no reflected light is detected in the third exposure period. The charge accumulated in the third exposure period represents noise due to background light. In contrast, in the first exposure period, in addition to the background light, charge generated by receiving the reflected light pulse is accumulated. Similarly, in the second exposure period, in addition to the background light, charge generated by receiving the reflected light pulse is accumulated.

[0023] The output voltage of the light receiving element due to the charge accumulated in the first exposure period is V1, the output voltage of the light receiving element due to the charge accumulated in the second exposure period is V2, and the output voltage of the light receiving element due to the charge accumulated in the third exposure period is V3. As in the example of Figure 1, if reflected light is detected in the first and second exposure periods but not in the third exposure period, then V1>V3. In the example of Figure 1, the time lengths of the three exposure periods are equal, so the background noise does not fluctuate in any of the exposure periods. In this case, the output voltage V3 in the third exposure period, in which no reflected light is detected, is taken as the background noise voltage V BG In the first exposure period and the second exposure period, both charges due to reflected light and charges due to background noise are accumulated. Therefore, the voltage V due to the charges accumulated by receiving reflected light in the first exposure period is Q1 can be expressed by the following equation (1). V Q1 =V1-V BG (1)

[0024] Similarly, the voltage V due to the charge accumulated by receiving the reflected light during the second exposure period Q2 can be expressed by the following equation (2). V Q2 =V2-V BG (2)

[0025] Since the time length of the first exposure period is equal to the time length of the second exposure period, the ratio of Q1 to Q2 is equal to the ratio of T0-Td to Td. That is, the ratio of T0-Td to Td is V Q1 and V Q2 Therefore, Td can be expressed by the following equation (3).

number

[0026] From equations (1), (2), and (3), Td can be expressed by the following equation (4).

number

[0027] On the one hand, as in the example shown in FIG. 2, when Td is longer than T0, the reflected light does not return during the first exposure period and returns during the second exposure period and the third exposure period, V1 < V3. In this case, the first exposure period accumulates only the charge due to background noise. On the other hand, in one or both of the second exposure period and the third exposure period, both the charge due to the reception of the reflected light pulse and the charge due to background noise are accumulated. In this case, Td can be expressed by the following formula (5).

Equation

[0028] Using the flight time Td calculated by formula (4) or (5), the distance D can be obtained by the operation of D = c × Td / 2 (c is the speed of light).

[0029] FIG. 3 is a diagram showing an example of three images generated based on the signals of the charges accumulated in each of the periods A0, A1, and A2 shown in FIG. 1, and a distance image generated from the data of these images. The light receiving device in this example is an image sensor including an array of a plurality of two-dimensionally arranged light receiving elements. A two-dimensional image is generated based on the signal of the charge accumulated in each light receiving element for each exposure period. In the example of FIG. 3, the reflected light pulse is detected in the first exposure period and the second exposure period, and only the noise component due to the background light is detected in the third exposure period. The distance of each pixel is obtained by the above operation using the pixel values obtained in each of the first to third exposure periods.

[0030] As described above, when there is noise due to background light other than the reflected light pulse, the pixel value of each pixel includes a noise component. In the above formulas (4) and (5), it is assumed that the charge accumulated in each light receiving element due to noise is equal in all exposure periods. However, in reality, the noise of each pixel varies for each exposure period. Simply performing the above calculation for each pixel may not be sufficient to remove the influence of noise.

[0031] Distance measuring devices generally output distance image data or three-dimensional point cloud data as shown in FIG. 3 as a result of distance calculation. Distance image data is expressed by a set (x, y, d) of horizontal position x, vertical position y, and distance d from a reference position, and represents the distance distribution of objects present in a scene. Three-dimensional point cloud data is data in which a plurality of points representing the distribution of objects in a scene are expressed in three-dimensional coordinates. The three-dimensional point cloud data can be generated, for example, by conversion from distance image data.

[0032] As described above, in the indirect ToF method, the distance is calculated by calculating the ratio of the charges accumulated in the light receiving element for each exposure period. Therefore, in a distance measuring device that outputs distance image data or 3D point cloud data, information on the reflectance of the object is lost. However, the reflectance information may be useful for the recognition process of the object. For example, such reflectance information may be useful in a control device that recognizes an object based on data transmitted from one or more distance measuring devices and controls a vehicle such as an autonomous vehicle based on the recognition result. As an example, when the distance image data or 3D point cloud data output from the distance measuring device contains a lot of noise, the accuracy of recognition may be improved by using brightness data that reflects the reflectance of the measurement point.

[0033] Based on the above considerations, the inventors have arrived at the configuration of the embodiment of the present disclosure described below.

[0034] A sensing device according to an embodiment of the present disclosure includes a light source, a light receiving device including at least one light receiving element that performs photoelectric conversion, and a processing circuit that controls the light source and the light receiving device. The processing circuit causes the light source to emit light toward a scene at least once, causes the light receiving device to receive light reflected by the light during each of a plurality of exposure periods, generates luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods based on light reception data from the light receiving device, the luminance data being used to generate distance data for the scene, and outputs the luminance data and timing data indicating the timing of each of the plurality of exposure periods.

[0035] According to the above configuration, the processing circuit generates luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods based on the light reception data from the light receiving device, the luminance data being used to generate distance data of the scene, and outputs the luminance data and timing data indicating the timing of each of the plurality of exposure periods. This enables the information processing device that has acquired the luminance data and the timing data to generate higher quality distance image data or 3D point cloud data based on the luminance data and the timing data.

[0036] The processing circuit may generate the distance data and switch between the distance data and the luminance data for output. According to the above configuration, it is possible to switch between a mode for outputting distance data and a mode for outputting luminance data and timing data as necessary. This allows flexible control, such as outputting luminance data with a large amount of data only when necessary.

[0037] The processing circuit may switch between the distance data and the luminance data and output the switched data in response to a request from an external device. The external device may be, for example, an information processing device that generates integrated distance data or point cloud data based on data output from a plurality of sensing devices.

[0038] The processing circuit may switch between outputting the distance data and the luminance data in accordance with a state of the received light data, which may include various states such as the amount of noise contained in the received light data or the magnitude of each value contained in the received light data.

[0039] The processing circuit may calculate a noise amount of the light reception data for at least one of the multiple exposure periods, output the luminance data when the noise amount exceeds a threshold, and output the distance data when the noise amount does not exceed the threshold. This makes it possible to output the luminance data instead of the distance data when the noise amount of the light reception data is large and the reliability of the distance data generated by the processing circuit is estimated to be low. The luminance data may be sent to an external information processing device having a processing capacity higher than that of the processing circuit, and processed by the information processing device. The information processing device can generate higher quality distance data or point cloud data based on the luminance data acquired from the sensing device.

[0040] The processing circuit may calculate a reflectance from the received light data for at least one of the exposure periods, and output the distance data when the reflectance exceeds a threshold, and output the luminance data when the reflectance does not exceed the threshold, thereby enabling control such that when the reflectance is high and reliable distance data can be generated, the distance data is output, and when not, the luminance data is output.

[0041] The processing circuit may repeat a plurality of frame operations, each of which may include causing the light source to emit the light toward the scene, causing the light receiving device to generate the light receiving data for each exposure period, and outputting at least one selected from the group consisting of a set of the luminance data and the timing data, and the distance data. With this configuration, the set of the luminance data and the timing data, or the distance data, may be repeatedly output at short time intervals, for example.

[0042] The processing circuit may determine whether to output the set of the luminance data and the timing data, or the distance data, for each frame operation, thereby making it possible to output an appropriate one of the set of the luminance data and the timing data, or the distance data, or both, for each frame operation.

[0043] When outputting the luminance data or the distance data, the processing circuit may output the luminance data or the distance data with an identifier indicating whether the luminance data or the distance data is included, so that another device performing processing based on the luminance data or the distance data can easily determine whether the acquired data includes the luminance data or the distance data.

[0044] The processing circuit may switch between outputting the distance data and the luminance data for each of a plurality of regions included in the scene, thereby enabling control such that when the reliability of distance data for only a portion of a scene is low, the luminance data for that portion is output.

[0045] The processing circuit may output fixed value data common to the plurality of frame operations when switching between outputting the set of the luminance data and the timing data and outputting the distance data.

[0046] According to another embodiment of the present disclosure, an information processing device includes a memory and a processing circuit, which acquires, from a sensing device, luminance data indicating a distribution of a reflected light amount of light reflected from a scene received during each of a plurality of exposure periods and timing data indicating a timing of each of the plurality of exposure periods, records the luminance data and the timing data in the memory, performs image processing on the luminance data, and generates first distance data based on the luminance data after the image processing and the timing data.

[0047] According to the above configuration, the processing circuit performs image processing on the luminance data, and generates first distance data based on the luminance data after the image processing and the timing data. The image processing may include, for example, a process of reducing noise in the luminance data.

[0048] The processing circuit may send a signal to the sensing device requesting it to switch between outputting second distance data generated within the sensing device based on the brightness data and the timing data, and outputting the brightness data.

[0049] The processing circuit may further acquire identification data indicating whether the second distance data or the brightness data has been output from the sensing device, and switch processing for the data output from the sensing device based on the identification data.

[0050] The processing circuit may identify a self-location of the sensing device, and when the self-location of the sensing device satisfies a predetermined condition, transmit a signal to the sensing device requesting output of the luminance data.

[0051] The processing circuit may determine an amount of noise in the luminance data, and when the amount of noise is greater than a reference value, transmit a signal to the sensing device requesting output of the luminance data.

[0052] A computer program according to another embodiment of the present disclosure causes a computer to perform the following operations. Make the light source emit light into the scene at least once. The light receiving device receives the reflected light from the light during each of a plurality of exposure periods. Based on the light reception data from the light receiving device, luminance data is generated that indicates a reflected light amount distribution corresponding to each of the multiple exposure periods, and is used to generate distance data of the scene. Outputting the luminance data and timing data indicating the timing of each of the plurality of exposure periods.

[0053] A computer program according to another embodiment of the present disclosure causes a computer to perform the following operations. From the sensing device, luminance data indicating a distribution of the amount of reflected light reflected from the scene during each of a plurality of exposure periods, and timing data indicating the timing of each of the plurality of exposure periods are obtained. Recording said luminance data and said timing data in said memory. Perform image processing on the luminance data. First distance data is generated based on the luminance data after the image processing and the timing data.

[0054] Hereinafter, exemplary embodiments of the present disclosure will be described in detail. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, arrangement and connection forms of components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly. Furthermore, in each figure, substantially the same components are given the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0055] <Embodiment 1> A system according to a first exemplary embodiment of the present disclosure will be described.

[0056] FIG. 4A is a block diagram showing the physical configuration of the system of this embodiment. This system includes a control device 200 and multiple ranging devices 100. The control device 200 is an information processing device that controls the operation of a vehicle, such as an autonomous vehicle. Each ranging device 100 may be a sensing device mounted on the vehicle. Each ranging device 100 is connected to the control device 200 by wire or wirelessly. Although the system of this embodiment includes multiple ranging devices 100, the number of ranging devices 100 may be one.

[0057] The distance measuring device 100 includes a light source 110, a light receiving device 120, a first processing circuit 130, a recording medium 170, and an input / output interface (IF) 150. The control device 200 includes a second processing circuit 230, a recording medium 270, and an input / output interface 210.

[0058] The light source 110 emits light toward a scene. The light receiving device 120 includes a sensor that detects light emitted from the light source 110 and reflected by an object. The first processing circuit 130 controls the light source 110 and the light receiving device 120 to perform operations based on the indirect ToF method described above. However, in this embodiment, the distance measuring device 100 does not perform distance calculation itself, but outputs luminance data of each measurement point that is the basis of distance calculation for each exposure period. In this manner, in this specification, a device that does not perform distance calculation itself but generates data necessary for distance calculation is referred to as a "distance measuring device". In addition to the luminance data of each pixel for each exposure period, the distance measuring device 100 outputs time data for specifying the start time and end time of each exposure period as timing data. These luminance data and time data are sent to the control device 200. The control device 200 uses the luminance data and timing data for each exposure period to calculate the distance of each measurement point by the above calculation. The control device 200 can generate data of a distance image or a three-dimensional point cloud based on the calculated distance. The control device 200 can further recognize specific objects in the scene based on the range image or 3D point cloud data, and control the operation of the vehicle's operating parts, such as the engine, steering, brakes, or accelerator, based on the recognition results.

[0059] The distance measuring device 100 outputs luminance data of a plurality of consecutive measurement points in a one-dimensionally or two-dimensionally expanding target area for each of a plurality of exposure periods. The light receiving device 120 may include an image sensor capable of acquiring a two-dimensional image. In this case, the distance measuring device 100 outputs luminance data of a plurality of two-dimensionally consecutive measurement points corresponding to a plurality of pixels of the image sensor. On the other hand, when the distance measuring device 100 is a sensing device that detects reflected light while changing the light emission direction one-dimensionally, the distance measuring device 100 outputs luminance data of a plurality of one-dimensionally consecutive measurement points. The distance measuring device 100 in this embodiment generates luminance data of a plurality of consecutive measurement points in a one-dimensionally or two-dimensionally expanding target area for each exposure period, and outputs the data together with timing data indicating the timing of each exposure period.

[0060] Next, a more specific configuration example of this embodiment will be described with reference to FIG. 4B. FIG. 4B is a block diagram showing a more detailed functional configuration of the distance measuring device 100 and the control device 200. In FIG. 4B, the specific configuration of only one distance measuring device 100 among the multiple distance measuring devices 100 is shown. The other distance measuring devices 100 may also have the same configuration. Note that the configuration may differ depending on the distance measuring device 100. For example, some distance measuring devices 100 may be configured to output general distance data.

[0061] [Configuration of distance measuring device] The distance measuring device 100 shown in Fig. 4B includes a light source 110, a light receiving device 120, a processing circuit 130, and an input / output interface 150. In the example of Fig. 4B, a clock 160 that outputs time data is provided outside the distance measuring device 100. The clock 160 outputs the time data to a plurality of distance measuring devices 100. The clock 160 may be provided inside the distance measuring device 100.

[0062] The light source 110 in this embodiment is an output unit of flash light that diffuses laser light over a wide range. The light source 110 includes, for example, a laser light source and a scattering plate, and emits light that diffuses over a wide range by scattering the laser light by the scattering plate.

[0063] The light receiving device 120 includes an image sensor 121 and an optical component (not shown). The optical component may include, for example, one or more lenses, and projects light from a certain range of angle of view onto the light receiving surface of the image sensor 121. The optical component may include other optical elements such as prisms or mirrors. The optical component may be designed such that light diffused from a point on an object in a scene is focused to a point on the light receiving surface of the image sensor 121.

[0064] The image sensor 121 is a sensor in which a plurality of light receiving elements 122 are two-dimensionally arranged along a light receiving surface. The image sensor 121 includes a plurality of light receiving elements 122, a plurality of charge accumulation units 124, and a plurality of switches 123. A plurality of (for example, three) charge accumulation units 124 are provided corresponding to each of the plurality of light receiving elements 122. The switch 123 is provided for each light receiving element 122, and switches the connection between the light receiving element 122 and the plurality of charge accumulation units 124 corresponding to the light receiving element 122. Each light receiving element 122 generates a charge according to the amount of light received by photoelectric conversion for each exposure period. Each charge accumulation unit 124 accumulates the charge generated by the light receiving element 122 during the corresponding exposure period. The number of charge accumulation units 124 corresponding to each light receiving element 122 is equal to or greater than the number of exposure periods required for distance measurement operation by indirect ToF. The switch 123 switches the connection between the light receiving element 122 and the charge accumulation unit 124 in response to switching of the exposure period in accordance with an instruction from the processing circuit 130. In the following description, a set of one light receiving element 122, the charge accumulation unit 124 corresponding to that one light receiving element 122, and the switch 123 corresponding to that one light receiving element 122 may be referred to as a "pixel."

[0065] The image sensor 121 may be, for example, a charge-coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, or an infrared array sensor. The image sensor 121 may have detection sensitivity not limited to the visible wavelength range, but may have detection sensitivity in wavelength ranges such as ultraviolet, near infrared, mid infrared, and far infrared. The image sensor 121 may be a sensor using a single photon avalanche diode (SPAD).

[0066] The image sensor 121 may have, for example, an electronic shutter system that exposes all pixels at once, that is, a global shutter mechanism. The electronic shutter may be a rolling shutter system that exposes each row, or an area shutter system that exposes only a part of the area that matches the irradiation range of the light beam. When the electronic shutter is a global shutter system, two-dimensional information can be acquired at one time by controlling the shutter in synchronization with the flash light. On the other hand, in the case of a system in which the exposure timing is changed for each part of the pixels, such as the rolling shutter system, the amount of information that can be acquired decreases because pixels with an inappropriate exposure timing cannot receive light. However, this problem can be solved by performing a process to correct the shift in the shutter timing for each pixel. As in a modified example described later, when the light source 110 is a beam scanner that emits a light beam with a small spread, the rolling shutter system may be more suitable than the global shutter system.

[0067] The light receiving device 120 receives light reflected from an object in a scene. The light receiving device 120 outputs data indicating the charge accumulated in each exposure period by the indirect ToF for all pixels of the image sensor 121 for each frame. In this embodiment, in the operation of one frame, light projection and exposure are repeated a common number of times for each exposure period so that a sufficient charge for distance calculation is accumulated in each of the multiple exposure periods. Then, at the stage where charge accumulation is completed in all exposure periods, the light receiving device 120 outputs data for all pixels in all exposure periods. The light receiving device 120 outputs the above data at a rate of, for example, 30 frames per second. The data output from the light receiving device 120 is recorded in a recording medium 170 such as a memory.

[0068] The recording medium 170 may include a memory such as a ROM or a RAM. The recording medium 170 records various data generated by the processing circuit 130. The recording medium 170 may further store a computer program executed by the processing circuit 130.

[0069] The processing circuit 130 determines the timing of the projection of the flash light by the light source 110 and the timing of the exposure of the light receiving device 120, and outputs an exposure control signal and a light projection control signal according to the timing. The processing circuit 130 also converts the charges accumulated in the charge accumulation section 124 for each pixel of the light receiving device 120 for each exposure period into pixel values ​​for each exposure period, and outputs array data of the pixel values, i.e., image data representing the luminance of each pixel. This image data for each exposure period is sent to the control device 200 via the interface 150.

[0070] The processing circuit 130 is an electronic circuit including a processor such as a CPU. The processing circuit 130 executes the processing in this embodiment by executing a program stored in a recording medium 170. The recording medium 170 may be included in the processing circuit 130.

[0071] A dotted frame 300 in FIG. 4B shows a schematic example of image data for each exposure period sent from the interface 150 to the control device 200. In this example, exposure is performed in three exposure periods A0, A1, and A2 shown in FIG. 1 and FIG. 2. The output data from the distance measuring device 100 in this example includes image data in a number equal to the number of exposure periods per frame. In this embodiment, the output data includes, in addition to the above image data, time data specifying each exposure period and data indicating the time length of the light pulse emitted from the light source 110, which are necessary for the control device 200 to perform distance calculations. The output data format will be described in detail later.

[0072] The clock 160 is a circuit that outputs detailed time information required for controlling the light source 110. The clock 160 measures time with nanosecond or microsecond accuracy, for example, and outputs the data. The clock 160 can be realized by an integrated circuit such as a real-time clock. The clock 160 may be synchronized with a time server. For the synchronization, a protocol such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) may be used. Alternatively, time synchronization may be performed based on the time of the control device 200 using GPS information. Note that the time synchronization method is not limited to the above and is arbitrary. Time synchronization allows the distance measuring device 100 to obtain accurate time data.

[0073] [Control device configuration] Next, the configuration of the control device 200 in this embodiment will be described. As shown in Fig. 4B, the control device 200 includes an input / output interface 210, a processing circuit 230, and a recording medium 270. The control device 200 controls the operation of an operating unit 240 in a vehicle. The operating unit 240 is a device that performs an operation related to automatic driving, such as an engine, a steering wheel, a brake, or an accelerator.

[0074] The interface 210 acquires output data from multiple distance measuring devices 100. The interface 210 also acquires map data distributed from an external server (not shown). The map data may be, for example, landmark map data. The interface 210 may be configured to acquire other data that the processing circuit 230 uses for processing. The other data may be, for example, color image, tilt, speed, or acceleration data acquired by the distance measuring device 100 or other sensors.

[0075] The processing circuit 230 includes a preprocessing unit 231, a point cloud data generating unit 232, an environment recognition unit 233, and an operation control unit 234. The processing circuit 230 is an electronic circuit including a processor such as a CPU or a GPU. The functions of the preprocessing unit 231, the point cloud data generating unit 232, the environment recognition unit 233, and the operation control unit 234 in the processing circuit 230 can be realized, for example, by the processor of the processing circuit 230 executing a program stored in the recording medium 270. In this case, the processor functions as the preprocessing unit 231, the point cloud data generating unit 232, the environment recognition unit 233, and the operation control unit 234. Each of these functional units may be realized by dedicated hardware. The recording medium 270 may be included in the processing circuit 230.

[0076] The pre-processing unit 231 performs image processing such as noise reduction on the output data from the distance measuring device 100 acquired via the interface 210 before distance calculation. This image processing is referred to as pre-processing in the following description. The noise reduction processing is performed on image data for each exposure period. Details of the noise reduction processing will be described later. Note that the pre-processing may include, in addition to the noise reduction processing, processes such as edge extraction or smoothing.

[0077] The point cloud data generator 232 performs the above-mentioned distance calculation for pixels at the same coordinates of each image, using the images for each exposure period processed by the preprocessor 231 and the time data for each exposure period of each ranging device 100. The point cloud data generator 232 calculates the distance for each pixel of the image sensor 121 of each ranging device 100, i.e., for each position on the xy coordinate system, and generates distance image data. After generating the distance image data for each ranging device 100, the point cloud data generator 232 converts the distance of each pixel of the distance image of each ranging device 100 into a point on the three-dimensional coordinate system that the control device 200 uses as a reference, based on the position and direction data of each ranging device 100.

[0078] The processing circuit 230 extracts objects such as automobiles, people, bicycles, etc. from the point cloud data in the unified coordinates generated by the point cloud data generation unit 232. The processing circuit 230 further matches the point cloud data in the unified coordinates with map data to recognize the state of its surroundings.

[0079] The operation control unit 234 determines the operation of the actuator 240 based on the position of the object in the three-dimensional space specified by the environment recognition unit 233, and transmits a control signal to the actuator 240.

[0080] The operation unit 240 executes an operation in response to a control signal transmitted from the operation control unit 234. For example, the operation unit 240 executes operations such as starting, accelerating, decelerating, stopping, and changing the direction of the vehicle.

[0081] [Operation of distance measuring device] Next, the operation of the distance measuring device 100 will be described.

[0082] Fig. 5 is a flowchart showing the operation of the distance measuring device 100. The processing circuit 130 of the distance measuring device 100 in this embodiment executes the operations of steps S1120 to S1190 shown in Fig. 5. The operation of each step will be described below.

[0083] (Step S1120) Processing circuit 130 first determines whether an operation end signal has been input from an external device (not shown). If an operation end signal has been input in step S1120, processing circuit 130 ends the operation. If an operation end signal has not been input in step S1120, the process proceeds to step S1130.

[0084] (Step S1130) The processing circuit 130 outputs a control signal to the light receiving device 120, and the light receiving device 120 opens the electronic shutter in accordance with the control signal output by the processing circuit 130. This starts a light detection operation for one frame.

[0085] (Step S1140) The processing circuit 130 determines whether all of the predetermined exposure periods necessary to generate one frame of distance data have been completed. In this embodiment, light projection and exposure are repeated a predetermined number of times in each exposure period. This causes charge to be accumulated in each pixel for all exposure periods. If charge accumulation has been completed in all exposure periods in step S1140, the process proceeds to step S1170. If charge accumulation has not been completed in all exposure periods in step S1140, the process proceeds to step S1150.

[0086] (Step S1150) The processing circuit 130 selects one of the predetermined exposure periods during which light projection and exposure have not yet been performed and charge has not yet been accumulated, and outputs a switching signal to the switch 123. This connects the light receiving element 122 in each pixel to the charge accumulation unit 124 that accumulates the charge during that exposure period. Details of the switching timing will be described later.

[0087] (Step S1160) The processing circuit 130 generates a signal for controlling the light projection timing of the light source 110 and the exposure timing of the light receiving device 120 according to a predetermined exposure start time and exposure end time for the selected exposure period with reference to the time data from the clock 160. The light source 110 emits a pulsed flash light of a predetermined length of time according to the control signal output from the processing circuit 130. The light receiving device 120 performs exposure according to a predetermined start time and end time of the exposure period based on the start time of light projection from the light source 110, and accumulates the charge generated by photoelectric conversion during the exposure period in the charge accumulation section 124 selected in step S1150. Details of the timing of light projection and exposure will be described later.

[0088] Steps S1140 to S1160 are repeated to complete the light receiving operation for one frame.

[0089] (Step S1170) If it is determined in step S1140 that light projection and light reception for the entire exposure period are completed, the processing circuit 130 closes the shutter in the light receiving device 120.

[0090] (Step S1180) Processing circuit 130 reads out the charges accumulated in each charge accumulation section 124 of each pixel during each exposure period by a series of operations from step S1130 to step S1160, converts the charges into pixel values, and records them on recording medium 170. Processing circuit 130 further erases the charges in each charge accumulation section 124.

[0091] (Step S1190) The processing circuit 130 converts the charge for each exposure period read out in step S1180 into a pixel value, and generates luminance image data for each exposure period. Furthermore, in addition to the luminance image data for each exposure period, the processing circuit 130 adds timing data for identifying each exposure period for each predetermined number of frames or at the beginning of each frame. The timing data may include, for example, information on the start time and end time of the exposure period. The processing circuit 130 outputs output data including the luminance image data for each exposure period and the timing data for identifying each exposure period via the interface 150. A specific example of the output data will be described later. After the output, the process returns to step S1120.

[0092] The processing circuit 130 repeats the operations from step S1120 to step S1190. This unit of repetition is sometimes called a "frame operation." By repeating a plurality of frame operations, data required for distance calculation is output for each frame. The output data is sent to the control device 200.

[0093] In this embodiment, in step S1190, output data including timing data for each exposure period is generated in addition to the luminance image data for each exposure period. The output data may include data on the light projection time, i.e., the time length of the pulse of the light beam.

[0094] [Example of light emission and reception operations] Among the operations of the distance measuring device 100 of this embodiment, a specific example of the operations of projection and exposure will be described.

[0095] FIG. 6 is a time chart showing an example of the operation of light projection and exposure by the distance measuring device 100. FIG. 6 shows an example of the operation of light projection and exposure for one frame on the time axis. From the top, the timing of light projection by the light source 110, the timing of exposure by the light receiving device 120, the shutter opening period of the light receiving device 120, and the timing of reading out the charge accumulation unit 124 are shown. The shutter of the light receiving device 120 is opened at the time of starting the light projection and exposure for one frame in step S1130, and is not closed until the exposure operation is completed. The operation for one frame is started by opening the shutter. In step S1150, one of the multiple exposure periods is selected, and in each pixel, the charge accumulation unit 124 corresponding to the exposure period and the light receiving element 122 are connected by the switch 123.

[0096] In this embodiment, the light source 110 repeatedly projects a pulse of flash light during each of the exposure periods A0, A1, and A2. The pulse duration T0 may be, for example, about 90 nanoseconds (nsec). With the start of light projection as the starting point of time, the charge storage unit 124 and the light receiving element 122 corresponding to the exposure period are connected by the switch 123 from the start to the end of the exposure period. An example of the timing of two consecutive light projections and exposures during each exposure period is shown within the dotted line frame at the top of FIG. 6.

[0097] In the exposure period A0, the timing of the start and end of exposure is the same as the timing of the start and end of light projection. For example, the exposure start point is 0 nsec and the exposure end point is 90 nsec, i.e., the same as the time length of the light projected pulse. The switch is connected at the same time as the light projection and disconnected at the same time as the light projection ends. Light projection by the light source 110, reception of light by the light receiving element 122, and accumulation of charge in the charge accumulation section 124 are repeated a predetermined number of times. Even if the energy of the reflected light is small in a single light projection, it can be made measurable by accumulating charges by receiving light multiple times.

[0098] When a predetermined number of light projections and exposures are completed in the exposure period A0, the next exposure period A1 is selected. In the newly selected exposure period A1, the charge accumulation unit 124 corresponding to the exposure period and the light receiving element 122 are connected by the switch 123. Even if the exposure period changes, the timing of projecting the flash light by the light source 110 does not change. On the other hand, in the newly selected exposure period A1, as shown in the enlarged view of FIG. 6, exposure starts when the pulse time length T0 (e.g., 90 nsec) has elapsed from the start of light projection, and exposure ends when T0 has elapsed after the start of exposure. That is, the switch 123 connects the charge accumulation unit 124 corresponding to the exposure period and the light receiving element 122 so that exposure starts simultaneously with the end of light projection and is performed for the same time length T0 as the pulse length of the light projection. As in the exposure period A0, the charge accumulation unit 124 holds the charge accumulated by the repeated exposure.

[0099] When a predetermined number of light projections and exposures are completed in the exposure period A1, light projections and exposures are further performed in the exposure period A2. In the exposure period A2, as shown in the enlarged view of FIG. 6, exposure starts 2T0 (e.g., 180 nsec) after the start of light projection, and ends T0 after the start of exposure. That is, exposure starts again when the light projection time length T0 has elapsed after the end of light projection. The exposure time length in the exposure period A2 is also T0, like the exposure periods A0 and A1. In the exposure period A2, a predetermined number of light projections and exposures are repeated, so that electric charges are accumulated in the charge accumulation section 124.

[0100] When charge accumulation is completed for all of the exposure periods A0, A1, and A2, the processing circuit 130 causes the light receiving device 120 to close the shutter (step S1170). Then, the processing circuit 130 reads out the charge corresponding to each exposure period accumulated in the charge accumulation unit 124 of each pixel. The processing circuit 130 generates and outputs luminance image data for each exposure period based on the read-out charge.

[0101] [Data format example] Next, an example of the format of data output by the distance measuring device 100 of this embodiment will be described.

[0102] 7A and 7B show an example of the output format of image data for each exposure period output from the interface 150 of the distance measuring device 100. The output data in this example includes fixed value data common to multiple frames and data that differs for each frame. As data common to multiple frames, for example, a fixed value is output once at the beginning of the output data or once for a predetermined number of frames.

[0103] The fixed values ​​include, for example, data of position, direction, angle of view, pixel arrangement, exposure period A0, exposure period A1, and exposure period A2. "Position" indicates the position of the image sensor 121 in the vehicle. The position may be, for example, 3-byte data expressed in three-dimensional coordinates with the center of the vehicle as the origin. "Direction" indicates the direction in which the light receiving surface of the surface of the image sensor 121 faces. The direction may be, for example, 3-byte data representing the normal vector of the light receiving surface expressed in three-dimensional coordinates with the center of the vehicle as the origin. "Angle of view" indicates the angle of view of the image sensor 121 and may be expressed, for example, by 2 bytes. "Pixel arrangement" indicates the number of pixels in each of the x and y directions of the image sensor 121 and may be, for example, 1 byte of data. "Exposure period A0", "exposure period A1", and "exposure period A2" indicate the time range of each exposure period. These time ranges may be, for example, expressed by 1 byte data describing the elapsed time from the start of the corresponding light projection in nanoseconds. Note that the number of exposure periods per frame may be a number other than 3. The number of exposure periods may also change during operation. For example, the number of exposure periods may be defined separately as a fixed value, and data on the time for each exposure period may be output.

[0104] The output data for each frame includes, for example, data of the date, time, luminance image of the exposure period A0, the luminance image of the exposure period A1, and the luminance image of the exposure period A2. The date is, for example, data indicating the year, month, and day, and can be expressed by 1 byte. The time is, for example, data indicating the hour, minute, second, millisecond, and microsecond, and can be expressed by 5 bytes. The luminance image of the exposure period A0 is a collection of pixel values ​​converted from the charge of each pixel accumulated during the exposure period A0, and can be expressed as, for example, 1 byte of data for each pixel. Similarly, the luminance image of the exposure period A1 and the luminance image of the exposure period A2 can be expressed as, for example, 1 byte of data for each pixel.

[0105] [Control device operation] Next, an example of the operation of the control device 200 will be described.

[0106] Fig. 8 is a flowchart showing the operation of the control device 200 in this embodiment. The processing circuit 230 of the control device 200 executes the operations of steps S2120 to S2200 shown in Fig. 8. The operation of each step will be described below.

[0107] (Step S2120) The processing circuit 230 judges whether or not an operation end signal has been input from an external device (not shown). If an operation end signal has been input, the operation ends. If an operation end signal has not been input, the process proceeds to step S2130.

[0108] (Step S2130) The processing circuit 230 judges whether or not there is an input of data from the distance measuring device 100. If there is an input of data from the distance measuring device 100, the process proceeds to step S2140. If there is no input of data from the distance measuring device 100, the process returns to step S2120.

[0109] (Step S2140) The processing circuit 230 performs pre-processing on the data acquired from the distance measuring device 100 to improve the accuracy of distance calculation. The pre-processing includes, for example, noise removal processing. Here, it is assumed that the distance measuring device 100 outputs data in the format shown in FIG. 7A and FIG. 7B. In this case, the pre-processing is performed on the image data corresponding to the exposure period A0, the image data corresponding to the exposure period A1, and the image data corresponding to the exposure period A2. The processing circuit 230 performs noise removal processing on the image data corresponding to each exposure period individually. As a method of noise removal processing, for example, adaptive filter processing using a Wiener filter may be performed.

[0110] The processing circuit 230 removes noise by removing high spatial frequency components in each image data through filter processing. The filter may be a filter other than a Wiener filter, for example, a Gaussian filter. In order to remove high spatial frequency components in each image data, a smoothing process may be performed by convolution operation using a predetermined filter, for example, a Laplacian filter. As pre-processing, a noise removal process other than an adaptive filter may be performed. Also, signal processing other than noise removal processing, such as contrast enhancement or edge extraction, may be performed.

[0111] (Step S2150) Processing circuitry 230 calculates the distance for each pixel using the image data for each exposure period that has been subjected to noise removal processing in step S2140. Processing circuitry 230 extracts pixel values ​​of the same pixel from the image data for each exposure period, calculates the time of flight based on the above-mentioned calculation formulas (4) and (5), and further calculates the distance.

[0112] (Step S2160) Processing circuit 230 further calculates the reflectance of the pixel whose distance has been calculated, using the distance for each pixel calculated in step S2150 and the image data for each exposure period on which noise has been removed in step S2140. The reflectance is the ratio of the total pixel value for each exposure period from which background noise has been removed to the value at 100% reflectance predetermined for each distance. The value at 100% reflectance can be obtained, for example, by measuring in advance the pixel values ​​of the reflected light from a reference white board for each distance from light receiving device 120. The value at 100% reflectance for each distance can be recorded in advance in recording medium 270 in the form of a table or the like.

[0113] 9 shows an example of a table recorded on the recording medium 270. The table in this example specifies the correspondence between the distance from the light receiving surface of the light receiving device 120 and the pixel value when reflected light from a virtual white board with a reflectance of 100% located at that distance is detected. The value of 100% reflectance may be recorded as a function of distance. The table or function specifying the correspondence between the distance and the value of 100% reflectance may be transmitted from the distance measuring device 100 to the control device 200.

[0114] (Step S2170) The processing circuit 230 converts the distance data for each pixel calculated in step S2150, that is, the distance image data, into three-dimensional point cloud data. Here, pixels of the distance image data whose distance value is 0 or infinity are not considered as points of point cloud data, and conversion is not performed. Only pixels of the distance image data whose valid distances are calculated are converted. The conversion is performed, for example, as follows. First, the distance between the origin of the coordinate system of the distance measuring device 100 and the origin of the coordinate system for data integration set in the control device 200 is calculated with reference to the data of the position of the distance measuring device 100 shown in Figs. 7A and 7B. Furthermore, the amount of rotation of the coordinate axes of the distance measuring device 100 relative to the coordinate axes of the integrated coordinate system of the control device 200 is calculated with reference to the data of the direction of the distance measuring device 100 shown in Figs. 7A and 7B. Based on the calculated distance and amount of rotation, coordinate conversion is performed for pixels having valid distance values ​​in the distance image generated in step S2150.

[0115] (Step S2180) The processing circuit 230 recognizes the environment around the distance measuring device 100 based on the point cloud data generated in step S2170 and the map data acquired from the outside. The environment can be recognized by matching the map data with the point cloud data. For example, the processing circuit 230 identifies a fixed object that matches a fixed object included in the map from the point cloud data, and identifies the positional relationship between the fixed object and the distance measuring device 100. Furthermore, the processing circuit 230 groups the point cloud data by object using the reflectance information for each point calculated in step S2160. This makes it possible to extract moving objects, such as people, animals, bicycles, or automobiles, that are in the vicinity of the distance measuring device 100 and are not included in the map.

[0116] Matching between map data and point cloud data can be performed, for example, as follows. First, a landmark map is referenced to select a landmark to be detected, and information such as the landmark's position coordinates is read. Then, based on the output data of the distance measuring device 100, the distance to the landmark is calculated. Matching can be performed based on the landmark's coordinates and the calculated distance to the landmark.

[0117] The pre-processed luminance image data and point cloud data are generated based on data output from the same distance measuring device 100, so there is no misalignment in posture. Therefore, the image data and point cloud data can be superimposed without distortion correction. The processing circuit 230 may identify the object by extracting an object that overlaps with the object obtained by grouping the point cloud from the luminance image data. Even if it is difficult to clarify the boundary of the object by grouping the point cloud alone, the boundary of the object becomes clearer by comparing it with the result of grouping pixels using a luminance image without misalignment. In this way, by integrating information obtained from both the luminance image data and the point cloud data, it is possible to more accurately reproduce the scene as a three-dimensional space. When extracting the object from the luminance image data, well-known image recognition processing such as object recognition by AI can be used. In addition, image processing such as edge extraction and contrast enhancement may be performed as pre-processing for the luminance image data.

[0118] (Step S2190) The processing circuit 230 determines the operation of the operating unit 240, such as the brake, accelerator, and steering wheel, based on the arrangement of structures such as buildings and moving objects in the three-dimensional space extracted in step S2180.

[0119] (Step S2200) The processing circuit 230 generates and outputs a control signal for controlling the actuator 240 based on the operation of the actuator 240 determined in step S2190.

[0120] [effect] As described above, the distance measuring device 100 of this embodiment generates luminance image data for each exposure period by performing light projection and exposure in each of a plurality of exposure periods. The control device 200 calculates the distance for each pixel based on the luminance image data for each exposure period, and further converts the distance data into three-dimensional point cloud data. Before calculating the distance, the control device 200 performs preprocessing such as noise removal on the luminance image data for each exposure period. This allows the distance to be calculated more accurately. Furthermore, by calculating the reflectance lost by the distance calculation, the accuracy of extraction or recognition of the object based on the point cloud data can be improved.

[0121] The configuration of this embodiment is merely an example, and various modifications of this embodiment are conceivable, some of which will be described below.

[0122] <Modification 1 of the First Embodiment> The light source 110 in the first embodiment emits a flash light that diffuses the laser light over a wide range, but the light source 110 may be configured to emit a light beam that is less diffuse than the flash light. By using a light beam such as a laser light, the energy density of the light can be made higher than when a flash light is used. Therefore, reflected light from a more distant object can be detected. In order to project light over a wide range, the light source 110 is controlled to change the direction of the light and project it multiple times.

[0123] [Configuration of distance measuring device] The configuration of the distance measuring device 100 in this modification is the same as that shown in Fig. 4B. However, the function and operation of the light source 110 are different. The configuration and operation of this modification will be described below, focusing on the differences from the first embodiment.

[0124] The light source 110 in this modification is a beam scanner capable of changing the emission direction of a light beam. The light source 110 sequentially irradiates a part of an area in a scene with a light beam in response to a command from the processing circuit 130. To achieve this function, the light source 110 includes a mechanism for changing the emission direction of the light beam.

[0125] FIG. 10 is a diagram showing an example of a light source 110. In this example, the light source 110 includes a light emitting element such as a laser and at least one movable mirror, for example, a MEMS mirror. Light emitted from the light emitting element is reflected by the movable mirror and directed toward a predetermined area in a scene. The processing circuit 130 can change the emission direction of the light beam by driving the movable mirror. This allows, for example, scanning a scene one-dimensionally or two-dimensionally with the light beam.

[0126] A light source capable of changing the light emission direction by a structure other than that having a movable mirror may be used. For example, a light source using a reflective waveguide as disclosed in Patent Document 4 may be used.

[0127] FIG. 11A is a perspective view showing a schematic example of a light source 110 using a reflective waveguide. For reference, mutually orthogonal X-axis, Y-axis, and Z-axis are shown schematic. The light source 110 includes an optical waveguide array 10A, a phase shifter array 20A, an optical splitter 30, and a substrate 40 on which they are integrated. The optical waveguide array 10A includes a plurality of optical waveguide elements 10 arranged in the Y direction. Each optical waveguide element 10 extends in the X direction. The phase shifter array 20A includes a plurality of phase shifters 20 arranged in the Y direction. Each phase shifter 20 includes an optical waveguide extending in the X direction. The plurality of optical waveguide elements 10 in the optical waveguide array 10A are connected to the plurality of phase shifters 20 in the phase shifter array 20A, respectively. The optical splitter 30 is connected to the phase shifter array 20A.

[0128] Light L0 emitted from a light-emitting element (not shown) is input to a plurality of phase shifters 20 in the phase shifter array 20A via an optical branching device 30. The light that has passed through the plurality of phase shifters 20 is input to each of the plurality of optical waveguide elements 10 with its phase shifted by a fixed amount in the Y direction. The light input to each of the plurality of optical waveguide elements 10 is output as a light beam L2 from a light output surface 10s parallel to the XY plane in a direction intersecting the light output surface 10s.

[0129] FIG. 11B is a diagram showing a schematic example of the structure of the optical waveguide element 10. The optical waveguide element 10 includes an optical waveguide layer 15 located between a first mirror 11 and a second mirror 12 facing each other, and a pair of electrodes 13 and 14 for applying a drive voltage to the optical waveguide layer 15. The optical waveguide layer 15 may be made of a material whose refractive index changes when a voltage is applied, such as a liquid crystal material or an electro-optical material. The transmittance of the first mirror 11 is higher than that of the second mirror 12. Each of the first mirror 11 and the second mirror 12 may be formed of a multilayer reflective film in which a plurality of high refractive index layers and a plurality of low refractive index layers are alternately stacked, for example.

[0130] The light input to the optical waveguide layer 15 propagates in the optical waveguide layer 15 along the X direction while being reflected by the first mirror 11 and the second mirror 12. The arrows in Fig. 11B show a schematic representation of how the light propagates. A part of the light propagating in the optical waveguide layer 15 is emitted to the outside through the first mirror 11.

[0131] By applying a driving voltage to the electrodes 13 and 14, the refractive index of the optical waveguide layer 15 changes, and the direction of the light emitted from the optical waveguide element 10 to the outside changes. The direction of the light beam L2 emitted from the optical waveguide array 10A changes according to the change in the driving voltage. Specifically, the emission direction of the light beam L2 shown in FIG. 11A can be changed along a first direction D1 parallel to the X-axis.

[0132] 11C is a diagram showing a schematic example of the phase shifter 20. The phase shifter 20 includes a total reflection waveguide 21 including a thermo-optical material whose refractive index changes with heat, a heater 22 in thermal contact with the total reflection waveguide 21, and a pair of electrodes 23 and 24 for applying a driving voltage to the heater 22. The refractive index of the total reflection waveguide 21 is higher than the refractive indexes of the heater 22, the substrate 40, and air. Due to the difference in refractive index, the light input to the total reflection waveguide 21 propagates along the X direction while being totally reflected within the total reflection waveguide 21.

[0133] By applying a driving voltage to the pair of electrodes 23 and 24, the total reflection waveguide 21 is heated by the heater 22. As a result, the refractive index of the total reflection waveguide 21 changes, and the phase of the light output from the end of the total reflection waveguide 21 is shifted. By changing the phase difference between the lights output from two adjacent phase shifters 20 among the multiple phase shifters 20 shown in Fig. 11A, the emission direction of the light beam L2 can be changed along a second direction D2 parallel to the Y axis.

[0134] With the above configuration, the light source 110 can change the emission direction of the light beam L2 two-dimensionally.

[0135] Details of the operation principle and operation method of the light source 110 as described above are disclosed, for example, in Patent Document 4. The disclosure of Patent Document 4 is incorporated herein by reference in its entirety.

[0136] In this modified example, the processing circuit 130 records, each time a light beam is projected, the charge accumulated when the light receiving elements in a portion of the image sensor 121 receive the light, in association with the time when the light beam was projected, on the recording medium 170.

[0137] 12 is a diagram showing an example of data that can be recorded on the recording medium 170. Each time the light beam is projected a specified number of times, the recording medium 170 stores a representative value of the time of projection in association with a time ID that indicates the representative value of the projection time. The recording medium 170 also stores the charge accumulated in the charge accumulation unit 124 in association with the projection of the light beam and the time ID corresponding to the projection for each light receiving element 122, i.e., for each pixel. Since the light beam is projected for each exposure, the charge and the time ID are recorded for each exposure.

[0138] [Operation of distance measuring device] Next, the operation of the distance measuring device 100 in this modified example will be described. Fig. 13 is a flowchart showing the operation of the distance measuring device 100 in this modified example. In the flowchart shown in Fig. 13, steps S3110 and S3120 are added to the flowchart shown in Fig. 5, and step S1180 is changed to step S3130. The operation of each step will be described below.

[0139] (Step S1120) Processing circuit 130 first determines whether an operation end signal has been input from an external device (not shown). If an operation end signal has been input in step S1120, processing circuit 130 ends the operation. If an operation end signal has not been input in step S1120, the process proceeds to step S3110.

[0140] (Step S3110) The processing circuit 130 judges whether or not the operation has been completed for all of the one or more light projection directions that are predetermined or specified by an external device (not shown). If it is judged that the operation has been completed for all of the light projection directions, the process proceeds to step S1190. If it is judged that the operation has not been completed for all of the light projection directions, the process proceeds to step S3120.

[0141] (Step S3120) The processing circuit 130 selects one of the one or more light-projecting directions that are determined in advance or that are specified by an external device (not shown) and that is not yet projecting light.

[0142] (Step S1130) The processing circuit 130 outputs a control signal to the light receiving device 120, and the light receiving device 120 opens the electronic shutter in accordance with the control signal output by the processing circuit 130. This starts a light detection operation for one light projection direction.

[0143] (Step S1140) Processing circuit 130 determines whether all of the predetermined exposure periods necessary to generate distance data have ended. If charge accumulation has ended in all exposure periods in step S1140, processing proceeds to step S1170. If charge accumulation has not ended in all exposure periods in step S1140, processing proceeds to step S1150.

[0144] (Step S1150) The processing circuit 130 selects one of the predetermined exposure periods during which light projection and exposure have not yet been performed and charge has not yet been accumulated, and outputs a switching signal to the switch 123.

[0145] (Step S1160) The processing circuit 130 generates a signal for controlling the light projection timing of the light source 110 and the exposure timing of the light receiving device 120 according to a predetermined exposure start time and exposure end time for the selected exposure period with reference to the time data from the clock 160. The light source 110 emits a pulsed light beam of a predetermined time length in the direction determined in step S3120 according to the control signal output by the processing circuit 130. The light receiving device 120 performs exposure according to the predetermined start time and end time of the exposure period based on the light projection start time of the light source 110, and accumulates the charge generated by photoelectric conversion during the exposure period in the charge accumulation unit 124 selected in step S1150.

[0146] (Step S1170) If it is determined in step S1140 that light projection and light reception for the entire exposure period are completed, the processing circuit 130 closes the shutter in the light receiving device 120.

[0147] (Step S3130) Processing circuit 130 reads out the electric charges accumulated during each exposure period by the series of operations from step S1130 to step S1160, converts the electric charges of the pixels that received reflected light among all the pixels into pixel values, and records the pixel values ​​for each pixel on recording medium 170.

[0148] (Step S1190) The processing circuit 130 generates luminance image data for each exposure period based on the data recorded in step S3130. Furthermore, the processing circuit 130 generates data on the projection time of the light beam in addition to the luminance image data for each exposure period. For example, the processing circuit 130 adds timing data for identifying each exposure period for each predetermined number of frames or at the beginning of each frame in addition to the luminance image data for each exposure period. The timing data may include, for example, information on the start time and end time of the exposure period. The processing circuit 130 outputs output data including the luminance image data for each exposure period, the timing data for identifying each exposure period, and the time data of each pixel via the interface 150. After output, the process returns to step S1120.

[0149] The processing circuit 130 repeats the operations from step S1120 to step S1190. The unit of this repetition is called a "frame operation." By repeating a plurality of frame operations, data required for distance calculation is output for each frame. The output data is sent to the control device 200.

[0150] [Data format example] 14A and 14B are diagrams showing an example of the output format of image data for each exposure period output from the distance measuring device 100 in this modified example. In this example, compared to the example shown in FIG. 7A and FIG. 7B, the number of light projection directions (e.g., 1 byte) is added to the fixed value. In addition, the data for each frame includes time data (e.g., 5 bytes) for each light projection direction in each exposure period. The data of the luminance value for each exposure period is added with data of a time ID (e.g., 1 byte) that specifies a representative value of the emission time of the light beam used to obtain the luminance value. The correspondence between the time ID and the actual time is recorded in the recording medium 170 as shown in FIG. 12. Data defining the correspondence between the time ID and the actual time is also recorded in advance in the recording medium 270 of the control device 200.

[0151] 15A and 15B are diagrams showing another example of the output format of image data for each exposure period output from the distance measuring device 100. In this example, a set of pixels that receive reflected light of a light beam emitted in the same direction is considered as one block, and data (for example, 4 bytes) that specifies the pixel area of ​​each block is included in the output data for each frame. Each block is, for example, a rectangular pixel area, and can be specified by the coordinates of the pixel located at the upper left and the pixel located at the lower right. The block data may include a block ID. In the example of FIG. 15B, after the date data, data of the pixel area of ​​the block is arranged as many times as the number of light projection directions, followed by data of the time for each light projection direction as many times as the product of the number of directions and the number of exposure periods. This is followed by image data in which the charge of each pixel accumulated in the exposure period A0 is converted into a pixel value, image data corresponding to the charge accumulated in the exposure period A1, and image data corresponding to the charge accumulated in the exposure period A2. When outputting information of a block as in this example, data specifying the pixel area of ​​the block may be recorded on the recording medium 170. Alternatively, data such as luminance values ​​may be recorded in the recording medium 170 for each group of blocks. In the data formats shown in Figs. 14B, 15A, and 15B, the time when the luminance data was acquired is added to each pixel or pixel block. That is, luminance image data for one frame includes luminance data at different times. The luminance image data, distance image data, and point cloud data generated from such a data format can be used by dividing them into pixels or pixel blocks included in a predetermined time range, rather than by frame. For example, when combining data acquired from the distance measuring device 100 with data including time information acquired from another measuring device, the control device 200 can divide the data into pixels or pixel blocks based on the time information and combine them.

[0152] <Modification 2 of the First Embodiment> Next, a second modification of the first embodiment will be described. In this modification, unlike the previous examples, the light source 110 is configured to emit a laser light beam with a small spread multiple times by changing the direction one-dimensionally. Unlike the previous examples, the light receiving device 120 is not an image sensor but includes a single or a small number of light receiving elements. The orientation of the light receiving device 120 is controlled according to the change in the light projection direction of the light source 110 so that the normal vector of the light receiving surface of the light receiving device 120 coincides with the light projection direction of the light source 110. Therefore, the processing circuit 130 records the data acquired by receiving light on the recording medium 170 for each beam projection direction, not for each light receiving element (pixel). The beam projection direction is specified, for example, by an angle from a reference position.

[0153] The operation of the distance measuring device 100 in this modification is the same as that of the distance measuring device 100 in the above-mentioned modification 1, and therefore the description will be omitted. For each light projection, luminance data corresponding to the charge accumulated by the light receiving device 120 in each exposure period is output as a data string for each frame. In this modification, one frame operation refers to a set of a series of light projection and exposure operations in multiple directions about one predetermined axis. In one frame operation, for example, a light beam can be sequentially emitted in a total of 720 directions by changing the direction by 0.5 degrees in a range of 360 degrees in the horizontal direction.

[0154] Figures 16A and 16B are diagrams showing an example of the output format of data output from the distance measuring device 100 in this modified example. Instead of the data of the angle of view and pixel arrangement in the example shown in Figures 14A and 14B, the fixed value in this modified example is, for example, 2 bytes, which represents the range of angles in the direction in which the light beam is emitted in one frame operation. The "direction" in this modified example is data that expresses the 0 degree direction preset in the distance measuring device 100 as a vector in a three-dimensional coordinate table with the center of the vehicle as the origin.

[0155] In this modification, for output data for each frame, first, the date (e.g., 1 byte) when the data for that frame was acquired is output. Next, a luminance value (e.g., 1 byte) and a time (e.g., 5 bytes) according to the charge accumulated in the exposure period A0 are repeatedly output for all light projection directions. Similarly, for the exposure periods A1 and A2, the luminance value and the time according to the charge accumulated are repeatedly output for all light projection directions.

[0156] 17A and 17B are diagrams showing another example of the output format of data output from the distance measuring device 100. In this example, for output data for each frame, first a date (e.g., 1 byte) is output, and then a time (e.g., 5 bytes) for each light projection direction in each exposure period is output. Next, a luminance value (e.g., 1 byte) according to the charge accumulated in exposure period A0 is repeatedly output for all light projection directions. Similarly, for exposure periods A1 and A2, luminance values ​​according to the accumulated charge are repeatedly output for all light projection directions.

[0157] <Embodiment 2> Next, a second exemplary embodiment of the present disclosure will be described.

[0158] In the first embodiment and its modifications, the distance measuring device 100 always outputs luminance data indicating the charge accumulated by receiving light in each exposure period without calculating the distance. In contrast, in this embodiment, the distance measuring device 100 calculates the distance for each pixel based on the luminance data, and can switch between outputting distance data and outputting luminance data depending on the situation. This output switching may be performed by an external instruction or by a determination based on internal data. After receiving data from the distance measuring device 100, the control device 200 determines whether the data is luminance data or distance data, and performs processing according to the type of data.

[0159] Fig. 18 is a diagram showing the functional configuration of a system in this embodiment. The hardware configuration in this embodiment is the same as the configuration shown in Fig. 4B, but the operation is different from that of embodiment 1. The configuration and operation of this embodiment will be described below, focusing on the differences from embodiment 1. In the following description, luminance data may be referred to as raw data.

[0160] [Configuration of distance measuring device] The processing circuit 130 in this embodiment has a function of calculating the distance for each pixel based on the raw data. The processing circuit 130 switches between outputting distance data and outputting raw data in response to a request from an external control device 200. Note that, although the system shown in FIG. 18 includes one distance measuring device 100, the system may include multiple distance measuring devices 100, as in the example shown in FIG. 4B.

[0161] Similar to the light source 110 in the first embodiment, the light source 110 in this embodiment outputs a flash of laser light that is diffused over a wide range.

[0162] The light receiving device 120 includes an image sensor 121 and optical components such as a lens. The image sensor 121 has a plurality of pixels arranged two-dimensionally. Each pixel includes a light receiving element 122, a plurality of charge accumulation units 124 corresponding to a plurality of exposure periods, and a switch 123 for switching the connection between the light receiving element 122 and each charge accumulation unit 124.

[0163] The processing circuit 130 determines the timing of projecting a flash of light from the light source 110 and the timing of exposing the light receiving device 120. Then, in accordance with the determined timing, the processing circuit 130 sends a light projection control signal to the light source 110 and an exposure control signal to the image sensor 121. The processing circuit 130 further calculates the distance for each pixel based on a signal generated for each exposure period by the charge accumulation unit 124. The processing circuit 130 outputs distance data indicating the calculated distance for each pixel, i.e., distance image data, via the interface 150. On the other hand, when the processing circuit 130 receives a signal requesting output of raw data from the control device 200, it outputs the raw data instead of the distance data.

[0164] [Control device configuration] The control device 200 has the same hardware configuration as the control device 200 shown in Fig. 4B. However, this embodiment differs from the first embodiment in that the processing circuit 230 transmits a signal requesting transmission of raw data to the distance measuring device 100 via the interface 210 depending on the situation. The processing circuit 230 transmits a signal requesting raw data to the distance measuring device 100 when more accurate data is required, for example, when the distance image data transmitted from the distance measuring device 100 contains a lot of noise.

[0165] In the configuration of FIG. 18, signals are directly transmitted and received between the interface 150 of the distance measuring device 100 and the interface 210 of the control device 200. The communication between the distance measuring device 100 and the control device 200 may be performed via a network such as the Internet. Another device on the network may be present between the distance measuring device 100 and the control device 200. The distance measuring device 100 or the control device 200 may communicate with a storage device such as a cloud server or a storage device via a network. For communication, a communication protocol such as http, ftp, TCP or UDP, and IP may be used. A PULL type communication method may be used, or a PUSH type communication method may be used. For wired transmission, for example, Ethernet, USB, RS-232C, HDMI (registered trademark), or coaxial cable may be used. Alternatively, for wireless transmission, any wireless communication method such as 3G / 4G / 5G defined by 3GPP and IEEE, wireless LAN, Wi-Fi, Bluetooth (registered trademark), or millimeter wave may be used.

[0166] [Operation of distance measuring device] Fig. 19 is a flowchart showing the operation of the processing circuit 130 of the distance measuring device 100 in this embodiment. In the flowchart shown in Fig. 19, step S1190 in the flowchart shown in Fig. 5 is replaced with steps S4110 to S4150. The operations from steps S1120 to S1170 are the same as those shown in Fig. 5. Below, differences from the operation shown in Fig. 5 will be described.

[0167] (Step S4110) When step S1180 is completed, the processing circuit 130 judges whether or not a signal requesting raw data output has been received from the control device 200. If the signal has been received, the process proceeds to step S4140. If the signal has not been received, the process proceeds to step S4120.

[0168] (Step S4120) The processing circuit 130 calculates the distance for each pixel based on the charge value of each pixel for each exposure period recorded on the recording medium 170. The method of calculating the distance is the same as the calculation method in step S2150 shown in FIG.

[0169] (Step S4130) Processing circuit 130 converts the distance for each pixel calculated in step S4120 into a pixel value to generate distance image data. Processing circuit 130 generates output data by adding an identifier to the distance image data, which indicates that the data is distance image data. Specific examples of the output data will be described later.

[0170] (Step S4140) If it is determined in step S4110 that a raw data output request has been made, the processing circuit 130 converts the charge values ​​for each exposure period recorded in the recording medium 170 into pixel values, and generates luminance image data for each exposure period. In this case, the processing circuit 130 generates output data by adding timing data indicating the timing of each exposure period to the luminance image data for each exposure period. The format of the output data is the same as that shown in Figures 7A and 7B, for example. However, an identifier indicating that the data is raw data, i.e., luminance image data, is added to the beginning of the output data.

[0171] (Step S4150) Processing circuit 130 outputs, via interface 150, the output data generated in step S4130 or step S4140.

[0172] Processing circuit 130 repeatedly executes the operations of steps S1120 to S4150. As a result, in response to a request from external control device 200, switching is performed between outputting distance image data and outputting luminance image data for each exposure period. Switching between outputting distance image data and outputting luminance image data for each exposure period can be performed at any frame. Alternatively, processing circuit 130 may switch the output format for every one or more predetermined number of frames. In that case, regardless of a request from control device 200, processing circuit 130 switches between outputting distance image data and outputting luminance image data for every predetermined number of frames.

[0173] [Data format example] Next, an example of the format of data output by the distance measuring device 100 in this embodiment will be described.

[0174] 20A and 20B are diagrams showing an example of the format of distance image data output from interface 150 of distance measuring device 100. In this example, an identifier indicating the data format, i.e., whether distance data or luminance data is included, is added to the beginning of a fixed value common to multiple frames. If the data format is distance data, the fixed value does not include the data of exposure periods A0 to A2 shown in FIG. 7B. As in the previous examples, the fixed value can be output, for example, once at the beginning of the output data, or before the first frame when switching the output data.

[0175] The output data for each frame includes the date (e.g., 1 byte) and time (e.g., 5 bytes) when the data for that frame was generated, and data on the distance of each pixel converted into a pixel value (e.g., 1 byte). A distance image is constructed from the pixel value data of multiple pixels.

[0176] Figures 21A and 21B are diagrams showing an example of the format of luminance image data output from the distance measuring device 100. The format of the luminance image data in this example is obtained by adding an identifier (for example, 1 byte) indicating that the data format is luminance image data to the beginning of the fixed value in the example shown in Figures 7A and 7B. All other points are the same as the example shown in Figures 7A and 7B.

[0177] 22A and 22B are diagrams showing another example of the format of the luminance image data output from the distance measuring device 100. In this example, data indicating the maximum pixel value (for example, 1 byte) previously measured for each distance is added as a fixed value to the format shown in FIG. 22A and FIG. 22B. This maximum pixel value represents the sum of pixel values ​​measured at one pixel during the exposure period A0 to A2, assuming that light is reflected from an object (for example, a white board) with a reflectance of 100% for each of a plurality of distances set in advance. This maximum pixel value may be recorded for a plurality of distances within a specific distance range (for example, 0.1 m to 50 m, etc.) so that, for example, the intervals become larger as the distance increases. The intervals may be determined so as to be proportional to the logarithm of the distance with base 2, for example. The data of the maximum pixel value is previously measured for each distance and is recorded in the recording medium 270 in the form of a table or the like, as shown in FIG. 9. The data of the maximum pixel value may be used when the processing circuit 230 in the control device 200 calculates the reflectance for each pixel, as described above. In the first embodiment and its modified examples, such maximum pixel value data may be included in the output data.

[0178] [Control device operation] Fig. 23 is a flowchart showing an example of the operation of the processing circuit 230 in the control device 200 in this embodiment. In the operation shown in Fig. 23, step S5110 is added between step S2130 and step S2140 in the flowchart shown in Fig. 8, and steps S5120 to S5140 are added between step S2170 and S2180. Other points are the same as the operation shown in Fig. 8. Below, the points different from the operation shown in Fig. 8 will be mainly described.

[0179] (Step S5110) When the processing circuit 230 receives data from the distance measuring device 100 in step S2130, it judges whether the data is raw data, i.e., luminance image data. This judgment is made based on the value of the data format in the fixed value of the input data. If the input data is raw data, it proceeds to step S2140. If the input data is not raw data, i.e., if it is distance image data, it proceeds to step S2170.

[0180] (Steps S2140 to S2160) The processes of steps S2140, S2150, and S2160 are the same as the processes of the corresponding steps in FIG. 8. In step S2140, the processing circuit 230 performs pre-processing, such as noise removal processing, on the raw data to improve the accuracy of distance calculation. In step S2150, the processing circuit 230 generates distance image data by calculating the distance for each pixel using the image data for each exposure period that has been subjected to noise removal processing in step S2140. In step S2160, the processing circuit 230 calculates the reflectance of the pixel whose distance has been calculated using the distance for each pixel calculated in step S2150 and the image data for each exposure period that has been subjected to noise removal processing in step S2140. Note that the value of 100% reflectance for each distance used to calculate the reflectance may be recorded in advance in the recording medium 170 as in the first embodiment, or may be included as a fixed value in the input data as shown in FIG. 22A and FIG. 22B.

[0181] (Step S2170) The processing circuit 230 converts into point cloud data the range image data generated in step S2150 or the range image data input from the distance measuring device 100. This conversion process is the same as the process in step S2170 in FIG.

[0182] (Step S5120) The processing circuit 230 estimates the self-location based on the point cloud data generated in step S2170 and the map data acquired from the outside. The self-location can be estimated, for example, by matching the map data with the point cloud data. For example, the processing circuit 230 identifies a fixed object that matches a fixed object included in the map from the point cloud data by matching. Then, the self-location can be estimated based on the value of the distance to the fixed object and the distance to the fixed object obtained from the map.

[0183] (Step S5130) The processing circuit 230 judges whether the position of the control device 200 estimated in step S5120 is in an area that meets a predetermined condition. The condition may be, for example, within 10 m of an intersection. If the position of the control device 200 meets the condition, the process proceeds to step S5140. If the position of the control device 200 does not meet the condition, the process proceeds to step S2190.

[0184] (Step S5140) The processing circuit 230 transmits a signal to the distance measuring device 100 via the interface 150, requesting the output of raw data. In step S5130, an example of a case where the estimated position of the control device 200 does not meet the condition is a case where the reliability of the distance image data input from the distance measuring device 100 is low. In such a case, the processing circuit 230 requests the distance measuring device 100 to output raw data instead of distance data. As a result, in the next frame, raw data is input from the distance measuring device 100 to the control device 200 instead of distance data.

[0185] The operations in the following steps S2180 to S2200 are the same as the corresponding operations in FIG.

[0186] In this embodiment, the distance measuring device 100 outputs either the distance image data or the luminance image data for each exposure period. Instead of this operation, when the distance measuring device 100 receives a signal requesting luminance image data, it may output both the distance image data and the luminance image data for each exposure period. Also, in this embodiment, the control device 200 always performs preprocessing and distance calculation when it receives luminance image data for each exposure period. Instead of this operation, the control device 200 may perform preprocessing and distance calculation only when necessary. The control device 200 may record and accumulate the acquired luminance image data for each exposure period in the recording medium 270. Alternatively, the luminance image data for each exposure period may be accumulated by transmitting it to an external device such as a server or storage via a network.

[0187] In this embodiment, the control device 200 determines whether or not to output a request signal for raw data in step S5130 based on its own position. Instead of such an operation, the control device 200 may determine whether or not to output a request signal for raw data based on the state of data acquired from the distance measuring device 100. For example, the request signal for raw data may be output when the value of a component with a high spatial frequency in the acquired distance image data or the proportion of high-frequency components in the whole exceeds a predetermined threshold. The request signal for raw data may be output when data acquired from a plurality of distance measuring devices 100 is inconsistent. Alternatively, the request signal for raw data may be output when the distance image data acquired from the distance measuring device 100 is inconsistent with the map data, such as when a point cloud is located inside a building indicated by the map data. For example, the request signal for raw data may be output when a predetermined number or more of points are in a position inconsistent with the map data among the point cloud corresponding to a plurality of pixels in the distance image data acquired from the distance measuring device 100. Alternatively, when the control device 200 acquires image data separately, if there are more than a predetermined number of points that cannot be matched between the image data and the distance image data, a request signal for raw data may be output.

[0188] The control device 200 may also acquire measurement results from a measurement device other than the distance measuring device 100, and determine whether or not to output a raw data request signal based on the measurement results from the other measurement device. The other measurement device may be a gyro sensor. For example, when the measurement result of the gyro sensor shows a steep change in the time direction, that is, when there is a large shaking or impact, the control device 200 may output a raw data request signal to the distance measuring device 100. Alternatively, when the measurement result of the gyro sensor shows an angle above or below a predetermined threshold, that is, when the entire system is tilted, such as when traveling on a steep slope, the control device 200 may output a raw data request signal to the distance measuring device 100. The other measurement device may also be a camera that captures a luminance image or a video. For example, a signal may be acquired from the camera, and when there is a pixel that shows a luminance exceeding a predetermined value in the acquired image or video frame, the control device 200 may output a raw data request signal to the distance measuring device 100. Such high luminance pixels may appear when strong sunlight is incident or when light from the headlights of an oncoming vehicle is incident. Alternatively, based on the analysis result of the motion vector obtained from the video, a raw data request signal may be output to the distance measuring device 100. For example, if the motion vector angle is unstable, it is estimated that the system is subjected to an impact or vibration, and therefore a raw data request signal may be output.

[0189] When the system is a moving body, the control device 200 may determine whether or not to output a raw data request signal based on an operation plan of the moving body. When the operation plan is a specific operation such as acceleration / deceleration, lane change, backing up, etc., the control device 200 may output a raw data request signal to the distance measuring device 100. Also, when a signal indicating an operation abnormality such as brake lock or traction abnormality is acquired from the sensors of various operating parts of the moving body, the control device 200 may output a raw data request signal to the distance measuring device 100.

[0190] The processing circuitry 230 of the control device 200 may determine whether or not to output a request signal for raw data based on the state of communication between the distance measuring device 100 and the control device 200. For example, when data exceeding a predetermined data amount is accumulated in a memory (not shown) included in the interface 210, the processing circuitry 230 may output a signal requesting distance image data. As an example, when the amount of data received by the interface 210 per second exceeds the predetermined data amount or is expected to exceed the predetermined data amount, the processing circuitry 230 may output a request signal for distance image data.

[0191] [effect] As described above, according to this embodiment, the processing circuit 130 of the distance measuring device 100 generates distance data for each pixel in addition to luminance data for each pixel for each exposure period. Then, the processing circuit 130 switches between outputting the distance data and the luminance data for each exposure period in response to a request from the external control device 200. The processing circuit 130 determines for each frame which of a set of luminance data for each exposure period and timing data indicating the timing of each exposure period, and the distance data, is to be output. Furthermore, the processing circuit 230 of the control device 200 requests the distance measuring device 100 to output luminance data when the data sent from the distance measuring device 100 does not satisfy a predetermined condition.

[0192] This operation realizes a system in which the distance measuring device 100 normally outputs distance data, and outputs luminance data for each exposure period only when more accurate distance measurement is required. The size of the luminance data for each exposure period is larger than the size of the distance data. According to the operation of this embodiment, the control device 200 can obtain the large-sized luminance data of each pixel for each exposure period only when necessary. This makes it possible to reduce the amount of communication between the control device 200 and the distance measuring device 100.

[0193] According to the operation of this embodiment, the distance measuring device 100 normally outputs distance image data, and in a place with complicated traffic conditions such as around an intersection, the distance measuring device 100 can output luminance data for each exposure period according to an instruction from the control device 200. The control device 200, which receives the luminance data for each exposure period from the distance measuring device 100, can generate more accurate distance information by preprocessing. In addition, the recognition performance can be improved by calculating the reflectance of each pixel, and the accuracy of environmental recognition can be improved. Furthermore, by recording detailed luminance data for each exposure period as it is or transferring it to another device such as a server via a network, data indicating the state of a complicated traffic environment can be retained. This makes it possible to record verification data that enables more detailed situation analysis, for example, when an accident occurs.

[0194] <Modification 1 of the second embodiment> Next, a first modification of the second embodiment will be described. In this modification, the light source 110 emits a light beam with a smaller spread instead of a flash light. The light source 110 is controlled to change the direction of the light and project the light multiple times in order to project the light over a wide range. The configuration of the light source 110 is the same as that in the first modification of the first embodiment.

[0195] Fig. 24 is a flowchart showing the operation of the distance measuring device 100 in this modification. In the operation shown in Fig. 24, steps S3110 and S3120 are added to the operation shown in Fig. 19, and step S1180 is replaced with step S3130. Other points are the same as the operation shown in Fig. 19. In addition, the operation shown in Fig. 24 is the operation shown in Fig. 13 with step S1190 replaced with steps S4110 to S4150. The operation of each step shown in Fig. 24 is the same as the operation of the corresponding step shown in Fig. 13 or Fig. 19, and therefore description thereof will be omitted.

[0196] As in this modification, by scanning a target scene with a light beam using light source 110 capable of changing the emission direction of the light beam, it is possible to obtain data required for measuring the distance to an object that is farther away than when using light source 110 that emits flash light. Also, the operations of steps S4110 to S4140 shown in Fig. 24 make it possible to switch between outputting distance data and outputting brightness data for each exposure period in response to a request from an external device.

[0197] The light source 110 and the light receiving device 120 may have a configuration similar to that of Modification 2 of the first embodiment. That is, a configuration may be adopted in which the light source 110 that emits a light beam and the light receiving device 120 that includes a single or a small number of light receiving elements are used, and the directions of the light source 110 and the light receiving device 120 are changed to obtain light receiving data required for distance measurement. Even in such a configuration, distance data and luminance data for each exposure period may be switched and output.

[0198] <Modification 2 of the second embodiment> Next, a second modification of the second embodiment will be described. In this modification, the contents of the output data are switched based on the processing results of the processing circuit 130 of the distance measuring device 100, not based on a request from an external device such as the control device 200. The configuration of this modification is the same as that of the second embodiment.

[0199] Fig. 25 is a flowchart showing the operation of the processing circuit 130 in the distance measuring device 100 in this modified example. In the operation shown in Fig. 25, steps S6110 and S6120 are executed instead of step S4110 in the operation shown in Fig. 19. The steps other than steps S6110 and S6120 are the same as the corresponding steps shown in Fig. 19. Below, the points that are different from the operation shown in Fig. 19 will be described.

[0200] (Step S6110) The processing circuit 130 processes the charge values ​​for each pixel in each exposure period read out from the charge storage unit 124 as two-dimensional array data according to the pixel array. Random noise can be extracted as high-frequency components of the spatial frequency in the image. In this modification, the processing circuit 130 first divides each of the luminance images of the exposure periods A0, A1, and A2 into a plurality of regions. For example, each luminance image can be divided into eight regions by dividing it into two in the y-axis direction and into four in the x-axis direction. Here, the luminance images of all of the exposure periods A0, A1, and A2 are divided in the same manner. The processing circuit 130 performs spatial frequency analysis on two-dimensional data indicating each divided region. For example, a two-dimensional Fourier transform can be used as a method of frequency analysis. The processing circuit 130 extracts the real part of each frequency component obtained as a result of the Fourier transform. The processing circuit 130 calculates the integral of the absolute value for components of a predetermined frequency or higher among the extracted frequency components, and determines this as the amount of noise. The processing circuit 130 calculates the above-mentioned amount of noise for each of the divided regions of the three luminance images corresponding to the exposure periods A0, A1, and A2, respectively.

[0201] (Step S6120) The processing circuit 130 evaluates the magnitude of noise based on the amount of noise in each divided region of each luminance image calculated in step S6110. If the amount of noise exceeds the threshold in step S6120, the process proceeds to step S4140. If the amount of noise does not exceed the threshold in step S6120, the process proceeds to step S4120. As an example of a method for evaluating the magnitude of noise, the processing circuit 130 may determine that there is a lot of noise when the sum of the amount of noise in the entire image is greater than a predetermined value in any of the luminance images of the exposure periods A0, A1, and A2. As another example of judging the magnitude of the noise amount, the processing circuit 130 may determine that there is a lot of noise when the amount of noise is greater than a predetermined value in some of the divided regions. For example, when the amount of noise is greater than a predetermined value in one of the eight divided regions, the processing circuit 130 may determine that there is a lot of noise. When the amount of noise is greater than a predetermined value in two or more regions, the processing circuit 130 may determine that there is a lot of noise. In this way, by dividing each luminance image into multiple regions and calculating and judging the amount of noise for each region, even if the noise is large in some regions of the image, it is possible to detect it without missing it.

[0202] In the above method, when it is determined that there is a lot of noise in the luminance image of any of the exposure periods A0, A1, and A2, the processing circuit 130 may determine that "the amount of noise exceeds the threshold value." The method of determining the amount of noise is not limited to the above method, and other methods may be used.

[0203] If it is determined in step S6120 that the amount of noise exceeds the threshold, processing circuit 130 outputs luminance image data, i.e., raw data, via interface 150 (S4140). On the other hand, if it is determined that the noise does not exceed the threshold, processing circuit 130 generates and outputs distance image data by calculating the distance for each pixel using the method described above (steps S4120 and S4130).

[0204] By repeating the operations from step S1120 to step S4150, the processing circuit 130 can switch between outputting distance image data and outputting luminance image data for each exposure period according to the state of the acquired luminance data. In this modification, the amount of noise that can be evaluated based on the magnitude of high-frequency components is used as the state of the luminance data, but the luminance data may be evaluated based on other indices related to the reliability of the luminance data. For example, a similar determination may be made based on the reflectance of each pixel calculated from the luminance data of each pixel and the data of the maximum luminance value when the reflectance of each distance is 100% that is recorded in advance. When the index value of the reflectance calculated based on the reflectance of one or more pixels in the three luminance images corresponding to the exposure periods A0, A1, and A2, respectively, is lower than a predetermined value, the processing circuit 130 may output the luminance image data for each exposure period instead of the distance image data. Alternatively, when a predetermined percentage of pixels whose luminance values ​​exceed a predetermined threshold value exist, the processing circuit 130 may output the luminance image data for each exposure period instead of the distance image data.

[0205] In this modification, in steps S6110 and S6120, the processing circuit 130 determines whether or not to output the luminance image data depending on the state of the acquired luminance data. Instead of such an operation, the processing circuit 130 may output the luminance image data for each exposure period when the distance measuring device 100 abnormally stops due to some cause. The abnormality of the distance measuring device 100 may be detected by another measuring device such as a gyro sensor.

[0206] The recording medium 170 stores the latest luminance image data of one or more frames together with time data so that the luminance image data can be output when the distance measuring device 100 stops abnormally. When the distance measuring device 100 stops abnormally, the processing circuit 130 reads out the luminance image data for each exposure period stored in the recording medium 170 going back from the time of the abnormal stop, and outputs it to the control device 200. The output destination may be not only the control device 200, but also an external system such as a traffic information center via communication.

[0207] In this modification, a light source 110 that emits a flash light is used, but as in the above-described modification 1, a beam scanner that emits a light beam with a small spread may be used as the light source 110. Also, as in modification 2 of the first embodiment, a light source that emits a light beam with a small spread and a light receiving device having a single or a small number of light receiving elements may be used.

[0208] The data formats in the above examples are merely examples, and the distance measuring device 100 may output similar information in other data formats. Furthermore, the distance measuring device 100 may generate luminance data or distance image data in any of the above formats, and then compress the data using a predetermined method to reduce the amount of data.

[0209] When multiple distance measuring devices 100 transmit data to one control device 200, an identifier for identifying the distance measuring device 100 may be added to the data included in the data format in each of the above examples.

[0210] When multiple distance measuring devices 100 output raw data to one control device 200, the control device 200 may jointly process the raw data from the multiple distance measuring devices 100. Such processing may include, for example, addition, averaging, filtering, etc. By jointly processing the raw data output from the multiple distance measuring devices 100, the accuracy of distance calculation can be improved.

[0211] <Modification 3 of the Second Embodiment> Next, a third modification of the second embodiment will be described. In this modification, the processing circuit 130 of the distance measuring device 100 determines whether to output luminance data or distance data for each region of an image, not for each frame. The processing circuit 130 in this modification divides the pixel group of the image sensor 121 into a plurality of regions, and switches between distance data and luminance data for each exposure period and outputs the data for each region. Therefore, the output data for each frame in this modification includes both raw data and distance data. The output data includes data for identifying each region and data indicating whether each region is output as raw data or distance data. After receiving data from the distance measuring device 100, the processing circuit 230 of the control device 200 similarly divides the image indicated by the data into a plurality of regions, determines whether each region is raw data or distance data, and performs different processing depending on the determination result.

[0212] The configuration of the system in this modified example is the same as that shown in Fig. 18. However, the data recorded on the recording medium 270 and the operations of the processing circuits 130 and 230 are different from those in the example of Fig. 18.

[0213] FIG. 26 is a diagram showing an example of data recorded on the recording medium 270. In this example, when the control device 200 receives image data from the distance measuring device 100, data as shown in FIG. 26 is recorded on the recording medium 270. In the example shown in FIG. 26, the date, the area ID, the range of the area, the area ID recorded for each pixel, the distance, the luminance in the exposure period A0, the luminance in the exposure period A1, and the luminance in the exposure period A3 are recorded. For pixels in which distance data is recorded, the luminance values ​​in the exposure periods A0, A1, and A2 are not recorded, and the pixels are blank. Conversely, for pixels in which the luminance values ​​in the exposure periods A0, A1, and A2 are recorded, the distances are not recorded, and the pixels are blank. After acquiring the data transmitted from the distance measuring device 100, the processing circuit 230 calculates and records the distances from the luminance value data in the exposure periods A0, A1, and A2 for pixels in which the distance values ​​are not recorded. As a result, as the process proceeds, the blank distance data is replaced by the calculated distance data in sequence.

[0214] [Operation of distance measuring device] Fig. 27 is a flowchart showing the operation of the distance measuring device 100 in this modified example. The operation shown in Fig. 27 is the same as the operation shown in Fig. 19 except that steps S4110 to S4150 in the operation shown in Fig. 19 are replaced with steps S7110 to S7170. Below, differences from the operation shown in Fig. 19 will be mainly described.

[0215] (Step S7110) When the operation of steps S1120 to S1180 completes acquisition of luminance data for each exposure period for one frame, the processing circuit 130 divides each image indicated by the data into multiple regions according to the region ID and region range information described in the signal transmitted from the control device 200.

[0216] (Step S7120) Processing circuit 130 determines whether or not output data generation has been completed for all of the regions divided in step S7110. If output data generation has been completed for all regions, processing proceeds to step S7140. If there are still regions for which output data generation has not been completed, processing proceeds to step S7130.

[0217] (Step S7130) Processing circuit 130 selects one of the regions divided in step S7110 for which output data has not yet been generated.

[0218] (Step S7140) The processing circuit 130 judges whether or not output of raw data is requested as output data for the region selected in step S7130 based on the signal transmitted from the control device 200. If output of raw data is requested for the region, the process returns to step S7120. If output of raw data is not requested for the region, the process proceeds to step S7150.

[0219] (Step S7150) For each pixel in the region selected in step S7130, processing circuit 130 calculates the distance based on the luminance value for each exposure period in the manner described above.

[0220] (Step S7160) Processing circuit 130 records in recording medium 170 the distances for each pixel calculated in step S7150.

[0221] 28 is a diagram showing an example of data recorded on the recording medium 170. A pair of a time ID and a detailed time, and a pair of an area ID and an area range are recorded. In addition, for each pixel, the ID of the area in which the pixel is included, the charge value in each exposure period read out in step S1180, and a time ID that identifies each exposure period are recorded. For pixels in areas where raw data output is not requested, the distance calculated in step S7150 is also recorded. For pixels in areas where raw data output is requested, the distance is not recorded.

[0222] (Step S7170) The processing circuit 130 converts the distance recorded for each pixel into a pixel value to generate distance data. For pixels without distance data, the processing circuit 130 converts the charge value for each exposure period into a brightness value to generate brightness data. The processing circuit 130 outputs the generated image data together with detailed time data via the interface 150.

[0223] [Data format example] 29A and 29B are diagrams showing an example of the format of output data. In this example, the position, direction, angle of view, pixel arrangement, and timing data of each exposure period of the distance measuring device are output as fixed values ​​common to multiple frames, similar to the example of FIG. 7A and FIG. 7B.

[0224] In the example of FIG. 29A and FIG. 29B, the following data is output in order as values ​​that vary for each frame. First, the date and detailed time when the data was acquired, and the number of regions into which the pixel array of the image sensor 121 is divided are output. Next, the start pixel coordinates and end pixel coordinates indicating the range of each region are output for the number of regions. In this modified example, the shape of each region is rectangular, but other shapes such as an ellipse may also be used. Furthermore, the number of regions for which distance data is output, the ID of each region for which distance data is output, the number of regions for which raw data is output, and the ID of each region for which raw data is output are followed. Following this information, the distances of each pixel for which raw data output is not requested are output. Furthermore, the luminance values ​​of each pixel for which raw data output is requested are output in order for each exposure period.

[0225] [Control device operation] Fig. 30 is a flowchart showing an example of processing executed by the processing circuit 230 of the control device 200 in this modified example. Among the steps shown in Fig. 30, steps S2120, S2130, and S2170-S2200 are the same as the corresponding steps shown in Fig. 8. The operation of each step will be described below.

[0226] (Step S2120) The processing circuit 230 judges whether or not an operation end signal has been input from an external device (not shown). If an operation end signal has been input, the operation ends. If an operation end signal has not been input, the process proceeds to step S2130.

[0227] (Step S2130) The processing circuit 230 determines whether or not there is data input from the distance measuring device 100. If there is data input from the distance measuring device 100, the processing circuit 230 records the data in the recording medium 270 and proceeds to step S9110. If there is no data input from the distance measuring device 100, the processing circuit 230 returns to step S2120.

[0228] (Step S9110) The processing circuit 230 obtains data indicating the number of regions when dividing an image indicated by the input data for each frame recorded on the recording medium 270, and the pixel range of each region. Furthermore, the processing circuit 230 judges whether or not the processing for all regions has been completed. If the processing for all regions has been completed, the process proceeds to step S9170 and step S2170. If there is an unprocessed region among the divided regions, the process proceeds to step S9120. Note that, in this modification, the processing from step S9170 to step S9190 and the processing from step S2170 to step S2200 are performed in parallel, but these may also be performed in series.

[0229] (Step S9120) The processing circuit 230 selects one of the divided regions that has not yet been processed.

[0230] (Step S9130) The processing circuit 230 refers to the input data for each frame recorded on the recording medium 270 and determines whether the area ID of the area selected in step S9120 is included in the distance area ID. If the area ID of the area is included in the distance area ID, the process returns to step S9110. If the area ID of the area is not included in the distance area ID, that is, if the area ID of the area is included in the Raw area ID, the process proceeds to step S9140.

[0231] (Step S9140) The processing circuit 230 acquires raw data of the region, that is, the luminance data of each of the exposure periods A0, A1, and A2 shown in FIG. 29A and FIG. 29B, from the input data from the distance measuring device 100 recorded on the recording medium 270. The processing circuit 230 performs pre-processing on the acquired raw data. An example of the pre-processing is noise removal processing. The noise removal is performed by using an adaptive filter on each of the luminance image of the exposure period A0, the luminance image of the exposure period A1, and the luminance image of the exposure period A2 of the pixel range extracted as the region. As the pre-processing, noise removal processing other than the adaptive filter may be performed. Also, signal processing other than the noise removal processing, such as contrast enhancement or edge extraction, may be performed.

[0232] (Step S9150) For the pixel range included in the region, the processing circuit 230 calculates the distance for each pixel using the image data for each exposure period that has been subjected to noise removal processing in step S9140. The pixel value of the same pixel is extracted from the image data for each exposure period within the region, and the distance can be calculated based on the above-mentioned calculation formula.

[0233] (Step S9160) Processing circuit 230 stores the distance data for each pixel in the region calculated in step S9150 in recording medium 270. After executing step S9160, the process returns to step S9110.

[0234] (Step S9170) Once distance data has been generated for all regions, processing circuit 230 generates distance image data by combining the distances of pixels having distance values ​​in the data input from distance measuring device 100 with the distances of pixels whose distances were calculated in step S9150. Then, the pixels of this distance image data are clustered according to distance. As a result, each pixel in the distance image data is classified into one of multiple clusters corresponding to multiple distance ranges.

[0235] (Step S9180) For each of the clusters generated in step S9170, excluding the clusters with a distance of 0 and a distance of infinity, the processing circuit 230 extracts a region around the cluster where the ratio N1 / N2 of the number of pixels N1 within the distance range characterizing the cluster to the number of pixels N2 outside the distance range characterizing the cluster is equal to or greater than a first threshold and less than a second threshold that is greater than the first threshold. The processing circuit 230 sets the rectangular region closest to the extracted region as the raw data request region.

[0236] (Step S9190) The processing circuit 230 divides the area other than the raw data request area set in step S9180 into one or more rectangular areas, and sets each rectangular area as a non-raw data request area, i.e., a distance data request area. The processing circuit 230 transmits to the distance measuring device 100, for each of the raw data request area and the distance data request area, an instruction signal including data indicating the range of the area and a data format specifying the area, i.e., a data code indicating whether it is raw data or distance data.

[0237] Fig. 31 is a diagram showing an example of the format of an instruction signal. In the example of Fig. 31, the instruction signal includes data indicating the number of areas, a data code (i.e., a binary value representing raw data / distance data), the range of each area, and the data code described in order of area ID.

[0238] Processing circuit 230 generates an output signal to distance measuring device 100 through operations from step S9170 to step S9190. In parallel, processing circuit 230 generates a control signal for the autonomous vehicle based on the output of distance measuring device 100 through operations from step S2170 to step S2200. The operations from step S2170 to step S2200 are the same as the operations of the corresponding steps shown in Fig. 8. After steps S9190 and S2200, the process returns to step S2120, and the same operations are repeated.

[0239] [effect] As described above, the processing circuit 230 of the control device 200 in this modification divides the pixel group in the image sensor 121 of the distance measuring device 100 into a plurality of regions based on the distance image data generated based on the output of the distance measuring device 100. Then, it requests the distance measuring device 100 to output raw data or distance data for each region. In response to the request from the control device 200, the processing circuit 130 of the distance measuring device 100 determines whether to output raw data or distance data for each region. This allows the processing circuit 230 of the control device 200 to perform more detailed signal processing by requesting the output of raw data for a region where the accuracy of the distance data output from the distance measuring device 100 is low, for example. As a result, the processing circuit 230 can generate highly accurate distance data that is difficult to generate in a short time by the processing circuit 130 of the distance measuring device 100, which has relatively low performance.

[0240] In this modification, the control device 200 requests the distance measuring device 100 to output raw data or distance data for each region. Instead of such an operation, for example, the control device 200 may output distance data for all pixels, and may request the distance measuring device 100 to output raw data in addition to distance data for a specific region. When requesting the output of additional raw data for a specific region, the instruction signal output from the control device 200 may include, for example, data specifying the number of regions for which additional raw data is to be output, and the range of each of those regions.

[0241] Fig. 32 is a flowchart showing an example of the operation of the distance measuring device 100 when an instruction signal is received requesting additional output of raw data for a specific area. In the operation shown in Fig. 32, steps S1110 to S1180, S7150 and S7160 are the same as the corresponding steps shown in Fig. 27. In the example shown in Fig. 32, steps S7210 and S7220 are executed instead of steps S7110, S7120, S7130 and S7240 in the example of Fig. 27. Also, step S7230 is executed instead of step S7170. These steps will be described below.

[0242] (Step S7210) After the operation of step S1180, the processing circuit 130 records in the recording medium 170 the area described in the instruction signal transmitted from the control device 200 for outputting the raw data in addition to the distance data.

[0243] (Step S7220) The processing circuit 130 determines whether or not distance calculation has been completed for all pixels of the image sensor 121. If distance calculation has been completed for all pixels, the process proceeds to step S7230. If there are pixels for which distance calculation has not been completed, the process performs the operations of steps S7150 and S7160, and then returns to step S7220.

[0244] (Step S7230) The processing circuit 130 outputs the distance data for each pixel recorded on the recording medium 170 and the raw data for each pixel in the area designated by the instruction signal sent from the control device 200 .

[0245] Figures 33A and 33B are diagrams showing an example of an output format in this modified example. In this example, following a fixed value, the date and time, the time, the distance calculated for all pixels, the number of regions for which raw data is output, the range of each region for which raw data is output, and the brightness indicating the charge accumulated in each of the exposure periods A0, A1, and A2 for the pixels in each region are output. In this way, when the distance measuring device 100 additionally outputs raw data for pixels in a specific region in addition to the distance information for all pixels, data in the format shown in Figures 33A and 33B can be output.

[0246] Fig. 34 is a diagram showing an example of the operation of the processing circuit 230 when the control device 200 receives data output in the format shown in Fig. 33A and Fig. 33B. The operation shown in Fig. 34 is similar to the operation shown in Fig. 30, except that steps S9110 to S9160 in the operation shown in Fig. 30 are replaced with steps S9310 to S9340. Below, the operation different from the example in Fig. 30 will be mainly described.

[0247] (Step S9310) Processing circuit 230 identifies the area with raw data output and its pixel range from the input data for each frame recorded on recording medium 270, and judges whether processing has been completed for all areas with raw data output. If processing has been completed for all areas with raw data output, proceed to step S9170 and step S2170. If there is an unprocessed area among the areas with raw data output, proceed to step S9320.

[0248] (Step S9320) The processing circuit 230 refers to the input data for each frame recorded on the recording medium 270, and selects one of the areas having raw data output that has not yet been processed.

[0249] (Step S9140) The processing circuit 230 acquires the luminance values ​​of the exposure periods A0, A1, and A2 for each pixel within the range of the area selected in step S9320. The processing circuit 230 performs pre-processing on the luminance values ​​of each acquired exposure period. The pre-processing is, for example, the noise removal processing described above.

[0250] (Step S9150) For the pixels included in the region, the processing circuit 230 calculates the distance for each pixel using the image data for each exposure period after the noise removal process in step S9140.

[0251] (Step S9330) Processing circuit 230 compares the variance of the distance data for each pixel output from distance measuring device 100 for the pixel range included in the region with the variance of the distance data calculated in step S9150 after performing noise removal processing in step S9140. If the variance of the distance data for each pixel calculated in step S9150 is smaller than the variance of the distance data output from the distance measuring device, proceed to step S9340. If the variance of the distance data for each pixel calculated in step S9150 is equal to or greater than the variance of the distance data output from the distance measuring device, return to step S9310.

[0252] (Step S9340) Processing circuit 230 replaces the distance value for each pixel in the region recorded on recording medium 170 with the distance value calculated in step S9150. This makes it possible to correct the distance data for the region to distance data with reduced noise.

[0253] In this modification, the control device 200 clusters the distance image for each distance range and divides the image into a plurality of regions based on the distribution state of distance data of peripheral pixels of the cluster, but the plurality of regions may be determined by other methods. The range of each region may be fixed and not changed for each frame. The regions may be set based on the distribution of distance data or the magnitude of noise. Alternatively, the setting of the regions may be changed according to the speed of a moving body on which the distance measuring device 100 and the control device 200 are mounted. Furthermore, the regions may be set by methods other than these.

[0254] When data of different data formats are mixed in one frame as in this modified example, data compression is difficult. Therefore, when compressing output data in which distance data and raw data are mixed in one frame, the distance data and raw data may be divided and compressed. The control device 200 that receives the data transmitted from the distance measuring device 100 may process the acquired data set in which distance data and raw data are mixed, and compress all pixels in the same format as distance data.

[0255] In this modification, the light source 110 of the distance measuring device 100 emits a flash light, but as in the first modification of the first embodiment, a beam scanner that emits a light beam with a small divergence angle may be used as the light source 110. When a beam scanner is used, the light projection direction from the light source 110 can be moved in the same manner as in the first modification of the first embodiment. The same operations as in the first modification of the first embodiment can be applied for the timing of light projection and reception, and the repetition of exposure associated with the scanning operation. As for the output data, detailed time data may be included for each direction of the light beam in order to deal with the shift in distance measurement time associated with scanning with a light beam with a small divergence angle to measure a wide range.

[0256] In the above embodiment, the operation by the indirect ToF method in which the light reception data is obtained in each of the three exposure periods has been mainly described, but the present disclosure is not limited to such an operation. For example, the number of exposure periods is not limited to three, and may be two or four or more. [Industrial Applicability]

[0257] The technology of the present disclosure can be widely used in devices or systems that perform distance measurement. For example, the technology of the present disclosure can be used as a component of a LiDAR (Light Detection and Ranging) system. [Explanation of symbols]

[0258] 100 Rangefinder 110 Light source 120 Photodetector 121 Image Sensor 122 Photodetector 123 Switch 124 Charge storage section 130 Processing circuit 150 Interface 160 Clock 170 Recording media 200 Vehicle control device 210 Interface 230 Processing circuit 270 Recording media 240 Operating Part

Claims

1. a light receiving device including at least one light receiving element that performs photoelectric conversion; A processing circuit for controlling the light receiving device; Equipped with The processing circuitry includes: causing the light receiving device to receive reflected light from a scene during each of a plurality of exposure periods; generating luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods based on the received light data from the light receiving device, the luminance data being used to generate distance data of the scene; outputting the luminance data and timing data indicating timings of each of the plurality of exposure periods; The processing circuitry further comprises: generating the distance data based on the luminance data; switching between the distance data and the luminance data and outputting the data; Sensing device.

2. the processing circuit switches between the distance data and the luminance data and outputs the data in response to a request from an external device. The sensing device according to claim 1 .

3. the processing circuit switches between the distance data and the luminance data and outputs the data in accordance with a state of the received light data. The sensing device according to claim 1 or 2.

4. The processing circuitry includes: Calculating a noise amount of the received light data for at least one of the plurality of exposure periods; When the amount of noise exceeds a threshold, the luminance data is output. When the amount of noise does not exceed the threshold, the distance data is output. A sensing device according to any one of claims 1 to 3.

5. The processing circuitry includes: calculating a reflectance from the received light data for at least one of the plurality of exposure periods; outputting the distance data when the reflectance exceeds a threshold value; When the reflectance does not exceed the threshold, output the luminance data. A sensing device according to any one of claims 1 to 3.

6. The processing circuit repeats a plurality of frame operations; Each of the plurality of frame actions comprises: causing the light receiving device to generate the light receiving data for each exposure period; outputting at least one selected from the group consisting of a set of the luminance data and the timing data, and the distance data; Including, the processing circuit determines, for each frame operation, whether to output the set of the luminance data and the timing data, or the distance data; A sensing device according to any one of claims 1 to 5.

7. The sensing device according to claim 6 , wherein when the processing circuit outputs the luminance data or the distance data, the processing circuit outputs the luminance data or the distance data by adding an identifier indicating whether the luminance data or the distance data is included.

8. The sensing device according to claim 1 , wherein the processing circuitry switches between outputting the distance data and outputting the luminance data for each of a plurality of regions included in the scene.

9. The sensing device according to claim 6 or 7, wherein the processing circuit outputs fixed value data common to the plurality of frame operations when switching between outputting the set of the brightness data and the timing data and outputting the distance data.

10. causing a light receiving device to receive reflected light from a scene during each of a plurality of exposure periods; generating luminance data indicating a reflected light amount distribution corresponding to each of the plurality of exposure periods based on the received light data from the light receiving device, the luminance data being used to generate distance data of the scene; outputting the luminance data and timing data indicating timings of each of the plurality of exposure periods; generating the distance data based on the luminance data; switching between outputting the distance data and the luminance data; A computer program that causes a computer to execute the following:

Images