Sensing devices

Abstract

To enable generation of higher-accuracy distance data.SOLUTION: A sensing device comprises an image sensor and a processing circuit. The processing circuit acquires pixel value data generated by the image sensor upon receiving reflected light from a scene in a plurality of exposure periods respectively, the pixel value data corresponding to the plurality of exposure periods, generates distance data based upon the pixel value data, and then outputs the distance data and the pixel value data.SELECTED DRAWING: Figure 4A

Description

[Technical Field]

[0001] This disclosure relates to a sensing device. [Background technology]

[0002] Conventionally, various devices have been proposed that acquire distance data of an object by illuminating it with light and detecting the reflected light from the object. The distance data of the target scene can be converted into, for example, 3D point cloud data and used. Point cloud data is typically data in which the distribution of points where objects exist in a scene is represented in 3D coordinates.

[0003] Patent Document 1 discloses a system that scans space with a light beam and detects reflected light from an object using a light sensor to acquire distance information to the object. This system generates and outputs information in which the measurement time is associated with each point in the point cloud data.

[0004] Patent Document 2 discloses an apparatus that measures the distance to structures surrounding a vehicle using a laser scanner and generates 3D point cloud data based on that distance data.

[0005] Patent Document 3 discloses a flash lidar system that is incorporated into a vehicle and measures the distance to an object using Time of Flight (ToF) technology.

[0006] Patent Document 4 discloses an apparatus that scans space with a light beam and generates distance data by receiving reflected light from objects with an image sensor. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2009-294128 [Patent Document 2] Japanese Patent Publication No. 2018-185228 [Patent Document 3] Japanese Patent Publication No. 2019-95452 [Patent Document 4] U.S. Patent Application Publication No. 2018 / 0217258 [Overview of the project] [Problems that the invention aims to solve]

[0008] One aspect of this disclosure provides a novel sensing device that outputs data necessary for distance measurement, and a novel information processing device that processes the data output from the sensing device. [Means for solving the problem]

[0009] A sensing device according to one aspect of the present disclosure comprises a light receiving device having at least one light receiving element that performs photoelectric conversion, and a processing circuit that controls the light receiving device. The processing circuit causes the light receiving device to receive reflected light from a scene during each of a plurality of exposure periods, generates luminance data that indicates the reflected light amount distribution corresponding to each of the plurality of exposure periods and is used to generate distance data of the scene based on the light receiving data from the light receiving device, and outputs the luminance data and timing data that indicates the timing of each of the plurality of exposure periods.

[0010] An information processing device according to another aspect of the present disclosure comprises a memory and a processing circuit. The processing circuit receives from a sensing device luminance data indicating the reflected light amount distribution of reflected light from a scene received during each of a plurality of exposure periods, and the timing of each of the plurality of exposure periods. The system acquires the indicated timing data, records the luminance data and the timing data in the memory, performs image processing on the luminance data, and generates first distance data based on the luminance data after image processing and the timing data.

[0011] The comprehensive or specific embodiments of this disclosure may be implemented by systems, apparatus, methods, integrated circuits, computer programs, or recording media such as computer-readable discs, or by any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. Computer-readable recording media may include volatile recording media or non-volatile recording media such as CD-ROMs (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. If an apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. In this specification and in the claims, “apparatus” may mean not only one device but also a system consisting of multiple devices. [Effects of the Invention]

[0012] According to one aspect of this disclosure, an information processing device can generate distance data with higher accuracy based on data output from a sensing device.

[0013] Additional benefits and advantages in the various embodiments contained herein will become apparent from this specification and the drawings. Each of these benefits and / or advantages may be provided individually by the various embodiments disclosed herein or by certain features in each embodiment. Not all features are required to obtain one or more of the benefits and / or advantages. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram illustrating an example of a distance measurement method using the indirect Time-of-Flight (TF) method. [Figure 2] This is a diagram illustrating an example of a distance measurement method using the indirect Time-of-Flight (TF) method. [Figure 3] This figure shows three images for each exposure period, and an example of a depth image generated from the data of those images. [Figure 4A]It is a block diagram showing the physical configuration of the system of Embodiment 1. [Figure 4B] It is a block diagram showing the functional configuration of the system of Embodiment 1. [Figure 5] It is a flowchart showing the operation of the distance measurement device. [Figure 6] It is a time chart showing an example of the operation of light projection and exposure by the distance measurement device. [Figure 7A] It is a first diagram showing an example of the output format of data output from the distance measurement device. [Figure 7B] It is a second diagram showing an example of the output format of data output from the distance measurement device. [Figure 8] It is a flowchart showing the operation of the control device in Embodiment 1. [Figure 9] It is a diagram showing an example of the correspondence relationship between distance and the maximum pixel value. [Figure 10] It is a diagram showing an example of a light source. [Figure 11A] It is a perspective view schematically showing an example of a light source using a reflective waveguide. [Figure 11B] It is a diagram schematically showing an example of the structure of an optical waveguide element. [Figure 11C] It is a diagram schematically showing an example of a phase shifter. [Figure 12] It is a diagram showing an example of data recorded on the recording medium 170. [Figure 13] It is a flowchart showing the operation of the distance measurement device in Modification 1 of Embodiment 1. [Figure 14A] It is a first diagram showing an example of the output format of data output from the distance measurement device in Modification 1 of Embodiment 1. [Figure 14B] It is a second diagram showing an example of the output format of data output from the distance measurement device in Modification 1 of Embodiment 1. [Figure 15A] It is a first diagram showing another example of the output format of data output from the distance measurement device. [Figure 15B] It is a second diagram showing another example of the output format of data output from the distance measurement device. [Figure 16A]The first figure shows yet another example of the output format of data output from a distance measuring device. [Figure 16B] The second figure shows yet another example of the output format of data output from a distance measuring device. [Figure 17A] The first figure shows yet another example of the output format of data output from a distance measuring device. [Figure 17B] The second figure shows yet another example of the output format of data output from a distance measuring device. [Figure 18] This figure shows the functional configuration of the system in Embodiment 2. [Figure 19] This is a flowchart showing the operation of the processing circuit of the distance measuring device in Embodiment 2. [Figure 20A] Figure 1 shows an example of the format of distance image data output from a distance measuring device. [Figure 20B] The second figure shows an example of the format of distance image data output from a distance measuring device. [Figure 21A] Figure 1 shows an example of the format of luminance image data output from a distance measuring device. [Figure 21B] The second figure shows an example of the format of luminance image data output from a distance measuring device. [Figure 22A] Figure 1 shows another example of the format of luminance image data output from a distance measuring device. [Figure 22B] The second figure shows another example of the format of luminance image data output from a distance measuring device. [Figure 23] This is a flowchart showing the operation of the control device in Embodiment 2. [Figure 24] This is a flowchart showing the operation of the distance measuring device in modified example 1 of Embodiment 2. [Figure 25] This is a flowchart showing the operation of the distance measuring device in modified example 2 of Embodiment 2. [Figure 26] This figure schematically shows an example of data recorded on the recording medium in a modified example 3 of Embodiment 2. [Figure 27]This is a flowchart showing the operation of the distance measuring device in modified example 3 of Embodiment 2. [Figure 28] This figure shows an example of data recorded on the recording medium in a modified example 3 of Embodiment 2. [Figure 29A] This figure shows an example of the output data format in modified example 3 of Embodiment 2. [Figure 29B] This figure shows an example of the output data format in modified example 3 of Embodiment 2. [Figure 30] This flowchart shows an example of the process performed by the control device in modified example 3 of Embodiment 2. [Figure 31] This figure shows an example of the format of an instruction signal. [Figure 32] This flowchart shows another example of the operation of a rangefinder. [Figure 33A] The first figure shows an example of the output format. [Figure 33B] The second figure shows an example of the output format. [Figure 34] This flowchart shows another example of the operation of the control device. [Modes for carrying out the invention]

[0015] In this disclosure, all or part of a circuit, unit, device, component or part, or b All or part of the functional blocks in a lock diagram can be implemented by one or more electronic circuits, including, for example, semiconductor devices, semiconductor integrated circuits (ICs), or LSIs (large-scale integrations). An LSI or IC may be integrated on a single chip or composed of multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. Here, we refer to them as LSIs or ICs, but the terminology may vary depending on the degree of integration; they may also be called system LSIs, VLSIs (very large-scale integrations), or ULSIs (ultra-large-scale integrations). Field-Programmable Gate Arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices that allow for the reconfiguration of internal junctions or the setup of internal circuit compartments within the LSI, can also be used for the same purpose.

[0016] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROMs, optical disks, or hard disk drives, and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0017] <Background> Before describing embodiments of this disclosure, we will describe examples of distance measuring methods that may be used in embodiments of this disclosure.

[0018] There are several methods for measuring distance to an object using a light source and a light-receiving device. For example, ToF (Time of Flight) techniques such as the direct ToF method and the indirect ToF method are commonly used. Of these, the direct ToF method calculates the distance to an object by directly measuring the time it takes for light to be emitted and return. On the other hand, the indirect ToF method measures the time it takes for light to be emitted and return by converting it into light intensity. These distance-measuring methods use a light source that emits light pulses and a light-receiving device equipped with one or more light-receiving elements. Below, as an example of a distance-measuring method, an example of a distance-measuring method using the indirect ToF method will be described.

[0019] Figures 1 and 2 illustrate an example of a distance measurement method using the indirect Time-of-Flight (TF) method. In Figures 1 and 2, the rectangular portions represent the periods of light pulse emission, the arrival of reflected light at the photodetector, and the three exposures. The horizontal axis represents time. Figure 1 shows an example where the light pulse is reflected from a relatively close object. Figure 2 shows an example where the light pulse is reflected from a relatively distant object. In Figures 1 and 2, waveform (a) shows the timing of light pulse emission from the light source, waveform (b) shows the period during which the reflected light of the light pulse reaches the photodetector, waveform (c) shows the first exposure period, waveform (d) shows the second exposure period, and waveform (e) shows the third exposure period. Let T0 be the time width of the light pulse used for distance measurement, and Td be the time from when the light pulse is emitted until it is received, i.e., the time of flight.

[0020] In this example, the first exposure period begins simultaneously with the start of light projection and ends simultaneously with the end of light projection. The second exposure period begins simultaneously with the end of light projection and ends when the same amount of time as the duration of the light pulse T0, i.e., the same amount of time as the first exposure period, has elapsed. The third exposure period begins simultaneously with the end of the second exposure period and ends when the same amount of time as the duration of the light pulse T0, i.e., the same amount of time as the first exposure period, has elapsed.

[0021] During the first exposure period, the reflected light that returns early is photoelectrically converted, and the resulting charge is accumulated. Q1 represents the energy of the light photoelectrically converted during the first exposure period. This energy Q1 is proportional to the amount of charge accumulated during the first exposure period. During the second exposure period, the reflected light that arrives after the end of the first exposure period and before time T0 has elapsed is photoelectrically converted, and the resulting charge is accumulated. Q2 represents the energy of the light photoelectrically converted during the second exposure period. This energy Q2 is proportional to the amount of charge accumulated during the second exposure period. During the third exposure period, the reflected light that arrives after the end of the second exposure period and before time T0 has elapsed is photoelectrically converted, and the resulting charge is accumulated. Q3 represents the energy of the light photoelectrically converted during the third exposure period. This energy Q3 is proportional to the amount of charge accumulated during the third exposure period.

[0022] Since the length of the first exposure period is equal to the time width T0 of the light pulse, in the example of Figure 1, the time width of the reflected light received during the second exposure period is equal to the time of flight Td. In the example of Figure 1, since the time of flight Td is shorter than the pulse time width T0, all of the reflected light returns to the photodetector by the end of the second exposure period. Therefore, no reflected light is detected during the third exposure period. The charge accumulated during the third exposure period represents noise due to background light. In contrast, during the first exposure period, in addition to background light, charge generated by the reception of the reflected light pulse is accumulated. Similarly, during the second exposure period, in addition to background light, charge generated by the reception of the reflected light pulse is accumulated.

[0023] Let the output voltage of the light-receiving element due to the charge accumulated during the first exposure period be V1, the output voltage of the light-receiving element due to the charge accumulated during the second exposure period be V2, and the output voltage of the light-receiving element due to the charge accumulated during the third exposure period be V3. As in the example of FIG. 1, when the reflected light is detected during the first exposure period and the second exposure period but not during the third exposure period, V1 > V3. In the example of FIG. 1, since the time lengths of the three exposure periods are equal, it is assumed that the background noise does not vary during all the exposure periods. In this case, the output voltage V3 of the third exposure period in which the reflected light is not detected can be taken as the voltage V of the background noise. and can be. During the first exposure period and the second exposure period, both the charge due to the reflected light and the charge due to the background noise are accumulated. Therefore, the voltage V Q1 due to the charge accumulated by the reception of the reflected light during the first exposure period can be expressed by the following equation (1). V Q1 = V1 - V BG (1)

[0024] Similarly, the voltage V Q2 due to the charge accumulated by the reception of the reflected light during the second exposure period can be expressed by the following equation (2). V Q2 = V2 - V BG (2)

[0025] Since the time length of the first exposure period is equal to the time length of the second exposure period, the ratio of Q1 to Q2 is equal to the ratio of T0 - Td to Td. That is, the ratio of T0 - Td to Td is equal to the ratio of V Q1 to V Q2 . Therefore, Td can be expressed by the following equation (3).

Number

[0026] From equations (1), (2), and (3), Td can be expressed by the following equation (4).

Number

[0027] On the other hand, as in the example shown in FIG. 2, when Td is longer than T0 and the reflected light does not return during the first exposure period but returns during the second and third exposure periods, V1<V3. In this case, the first exposure period accumulates only the charge due to background noise. On the other hand, in one or both of the second and third exposure periods, both the charge due to the reception of the reflected light pulse and the charge due to background noise are accumulated. In this case, Td can be expressed by the following formula (5). Using the flight time Td calculated by formula (4) or (5), the distance D can be obtained by the calculation of D = c×Td / 2 (where c is the speed of light).

Equation

[0028] Using the flight time Td calculated by formula (4) or (5), the distance D can be obtained by the calculation of D = c×Td / 2 (where c is the speed of light).

[0029] FIG. 3 is a diagram showing an example of three images generated based on the charge signals accumulated in each of the periods A0, A1, and A2 shown in FIG. 1, and a distance image generated from the data of those images. The light receiving device in this example is an image sensor including an array of a plurality of two-dimensionally arranged light receiving elements. For each exposure period, a two-dimensional image is generated based on the signal of the charge accumulated in each light receiving element. In the example of FIG. 3, the reflected light pulse is detected in the first and second exposure periods, and only the noise component due to background light is detected in the third exposure period. The distance of each pixel is obtained by the above calculation using the pixel values obtained in each of the first to third exposure periods.

[0030] As described above, if there is noise from background light other than reflected light pulses, the pixel value of each pixel will include a noise component. Equations (4) and (5) above assume that the charge accumulated on each photodetector due to noise is the same throughout the entire exposure period. However, in reality, the noise of each pixel varies from one exposure period to the next. Performing the above calculations for each pixel alone may not be sufficient to remove the effects of noise.

[0031] Distancing devices generally output distance image data or 3D point cloud data as a result of distance calculation, as shown in Figure 3. Distance image data is represented by a set of (x,y,d) consisting of horizontal position x, vertical position y, and distance d from a reference position, and represents the distance distribution of objects present in the scene. 3D point cloud data is data that represents the distribution of objects in the scene using 3D coordinates for multiple points. 3D point cloud data can be generated, for example, by conversion from distance image data.

[0032] As described above, in the indirect ToF method, distance is calculated by determining the ratio of the charge accumulated on the photodetector for each exposure period. Therefore, in distance measuring devices that output distance image data or 3D point cloud data, information on the reflectance of the object is lost. However, reflectance information can be useful in object recognition processing. For example, such reflectance information can be useful in a control device that recognizes an object based on data transmitted from one or more distance measuring devices and controls a vehicle such as an autonomous vehicle based on the recognition result. As an example, if the distance image data or 3D point cloud data output from the distance measuring device contains a lot of noise, it may be possible to improve the accuracy of recognition by using luminance data that reflects the reflectance of the measurement point.

[0033] Based on the above considerations, the inventors have come up with the configuration of the embodiments of the present disclosure described below.

[0034] A sensing device according to one embodiment of the present disclosure includes a light source, a light receiving device equipped with at least one photoreceiving element that performs photoelectric conversion, and a processing circuit that controls the light source and the light receiving device. The processing circuit includes the following: The processing circuit causes the light source to emit light toward the scene at least once, causes the light receiving device to receive the reflected light from the light for each of the plurality of exposure periods, generates luminance data that indicates the reflected light amount distribution corresponding to each of the plurality of exposure periods and is used to generate the scene distance data, and outputs the luminance data and timing data that indicates the timing of each of the plurality of exposure periods.

[0035] According to the above configuration, the processing circuit generates luminance data that indicates the reflected light quantity distribution corresponding to each of the plurality of exposure periods, based on the light received data from the light receiving device, and is used to generate the scene distance data. The circuit outputs the luminance data and timing data that indicates the timing of each of the plurality of exposure periods. As a result, an information processing device that has acquired the luminance data and timing data can generate higher quality distance image data or 3D point cloud data based on the luminance data and timing data.

[0036] The processing circuit may generate the distance data and switch between outputting the distance data and the brightness data. With the above configuration, it is possible to switch between a mode that outputs distance data and a mode that outputs brightness data and timing data as needed. This allows for flexible control, such as outputting brightness data with a large amount of data only when necessary.

[0037] The processing circuit may switch between outputting the distance data and the brightness data in response to a request from an external device. The external device may be, for example, an information processing device that generates integrated distance data or point cloud data based on data output from multiple sensing devices.

[0038] The processing circuit may switch between outputting the distance data and the brightness data depending on the state of the received light data. The state of the received light data may include various states, such as the amount of noise contained in the received light data, or the magnitude of each value contained in the received light data.

[0039] The processing circuit may calculate the amount of noise in the received light data for at least one of the plurality of exposure periods, and output the luminance data when the amount of noise exceeds a threshold, and output the distance data when the amount of noise does not exceed the threshold. This allows luminance data to be output instead of distance data when the amount of noise in the received light data is large and the reliability of the distance data generated by the processing circuit is estimated to be low. The luminance data may be sent to an external information processing device with higher processing capabilities than the processing circuit and processed by the information processing device. The information processing device can generate higher quality distance data or point cloud data based on the luminance data acquired from the sensing device.

[0040] The processing circuit may calculate the reflectance from the light-receiving data for at least one of the plurality of exposure periods, output the distance data when the reflectance exceeds a threshold, and output the luminance data when the reflectance does not exceed the threshold. This makes it possible to control the output so that distance data is output when the reflectance is high and reliable distance data can be generated, and luminance data is output otherwise.

[0041] The processing circuit may repeat a plurality of frame operations. Each of the plurality of frame operations involves causing the light source to emit the light toward the scene, causing the light receiving device to generate the light reception data for each exposure period, and outputting at least one selected from the group consisting of a pair of luminance data and timing data, and distance data. This configuration may include the following: With this configuration, a pair of luminance data and timing data, or distance data, can be repeatedly output, for example, at short time intervals.

[0042] The processing circuit may decide for each frame operation whether to output the pair of luminance data and timing data, or the distance data. This allows for the output of either the pair of luminance data and timing data, or the distance data, or both, for each frame operation.

[0043] When the processing circuit outputs the luminance data or the distance data, it may output the luminance data or the distance data with an identifier indicating which of the two types of data is included. This allows other devices that perform processing based on the luminance data or distance data to easily determine which of the two types of data is included in the acquired data.

[0044] The processing circuit may switch between outputting the distance data and the brightness data for each of the multiple regions included in the scene. This makes it possible to control the output so that if the reliability of the distance data is low for only a portion of the scene, brightness data is output only for that region.

[0045] When the processing circuit switches between outputting the pair of luminance data and timing data and outputting the distance data, it may output fixed value data common to the multiple frame operations.

[0046] An information processing device according to another embodiment of the present disclosure comprises a memory and a processing circuit. The processing circuit acquires from a sensing device luminance data indicating the reflected light amount distribution of reflected light from a scene received during each of a plurality of exposure periods, and timing data indicating the timing of each of the plurality of exposure periods, records the luminance data and the timing data in the memory, performs image processing on the luminance data, and generates first distance data based on the luminance data after image processing and the timing data.

[0047] According to the above configuration, the processing circuit performs image processing on the luminance data and generates first distance data based on the luminance data after image processing and the timing data. The image processing may include, for example, a process to reduce noise in the luminance data.

[0048] The processing circuit may transmit a signal to the sensing device requesting it to switch between outputting a second distance data, which is generated internally within the sensing device based on the brightness data and the timing data, and outputting the brightness data.

[0049] The processing circuit may further acquire identification data from the sensing device indicating whether the second distance data or the brightness data was output, and may switch the processing of the data output from the sensing device based on the identification data.

[0050] The processing circuit may determine the position of the sensing device, and when the position of the sensing device satisfies predetermined conditions, it may transmit a signal to the sensing device requesting the output of the brightness data.

[0051] The processing circuit may determine the amount of noise in the brightness data, and if the amount of noise is greater than a reference value, it may send a signal to the sensing device requesting the output of the brightness data.

[0052] Computer programs according to other embodiments of this disclosure cause a computer to perform the following actions: • The light source should emit light towards the scene at least once. The light receiving device receives the reflected light from the aforementioned light during each of the multiple exposure periods. Based on the light received data from the light receiving device, luminance data is generated that shows the reflected light quantity distribution corresponding to each of the plurality of exposure periods, and is used to generate the distance data of the scene. The system outputs the luminance data and timing data indicating the timing of each of the multiple exposure periods.

[0053] Computer programs according to other embodiments of this disclosure cause a computer to perform the following actions: The sensing device acquires luminance data indicating the distribution of reflected light from the scene received during each of the multiple exposure periods, and timing data indicating the timing of each of the multiple exposure periods. The brightness data and timing data are recorded in the memory. • Perform image processing on the aforementioned brightness data. Based on the brightness data after image processing and the timing data, a first distance data is generated.

[0054] The following describes exemplary embodiments of this disclosure. The embodiments described below are either comprehensive or specific examples. The numerical values, shapes, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in the independent claim representing the highest-level concept are described as optional components. Also, the figures are schematic diagrams and not necessarily strictly illustrative. In addition, substantially identical components are denoted by the same reference numerals in the figures, and redundant explanations may be omitted or simplified.

[0055] <Embodiment 1> The system relating to Exemplary Embodiment 1 of this Disclosure will be described.

[0056] Figure 4A is a block diagram showing the physical configuration of the system according to this embodiment. The system comprises a control device 200 and a plurality of distance measuring devices 100. The control device 200 is an information processing device that controls the operation of a vehicle, such as an autonomous vehicle. Each distance measuring device 100 may be a sensing device mounted on the vehicle. Each distance measuring device 100 is connected to the control device 200 by wire or wireless. Although the system according to this embodiment comprises a plurality of distance measuring devices 100, the number of distance measuring devices 100 may be one or less.

[0057] The distance measuring device 100 includes a light source 110, a light receiving device 120, a first processing circuit 130, a recording medium 170, and an input / output interface (IF) 150. The control device 200 includes a second processing circuit 230, a recording medium 270, and an input / output interface 210.

[0058] The light source 110 emits light toward the scene. The light receiving device 120 includes a sensor that detects light emitted from the light source 110 and reflected by an object. The first processing circuit 130 controls the light source 110 and the light receiving device 120 to perform the operation based on the indirect ToF method described above. However, in this embodiment, the distance measuring device 100 does not perform the distance calculation itself, but outputs luminance data for each measurement point that is the basis for the distance calculation for each exposure period. Thus, in this specification, a device that does not perform the distance calculation itself but generates the data necessary for the distance calculation is called a "distance measuring device". This is referred to as [the name of the device]. The distance measuring device 100 outputs timing data, which includes brightness data for each pixel for each exposure period, as well as time data to identify the start and end times of each exposure period. This brightness data and time data are sent to the control device 200. The control device 200 uses the brightness data and timing data for each exposure period to calculate the distance to each measurement point using the above calculation. Based on the calculated distance, the control device 200 can generate a distance image or 3D point cloud data. Furthermore, based on the distance image or 3D point cloud data, the control device 200 can recognize a specific object in the scene and control the operation of the vehicle's operating parts, such as the engine, steering, brakes, or accelerator, based on the recognition result.

[0059] The distance measuring device 100 outputs luminance data for multiple consecutive measurement points in a target area that extends in one or two dimensions for each of the multiple exposure periods. The light receiving device 120 may include an image sensor capable of acquiring a two-dimensional image. In that case, the distance measuring device 100 outputs luminance data for multiple consecutive measurement points corresponding to multiple pixels of the image sensor. On the other hand, if the distance measuring device 100 is a sensing device that detects reflected light while changing the direction of light emission in one dimension, the distance measuring device 100 outputs luminance data for multiple consecutive measurement points in one dimension. In this embodiment, the distance measuring device 100 generates luminance data for multiple consecutive measurement points in a target area that extends in one or two dimensions for each exposure period and outputs it together with timing data indicating the timing of each exposure period.

[0060] Next, a more specific configuration example of this embodiment will be described with reference to Figure 4B. Figure 4B is a block diagram showing a more detailed functional configuration of the distance measuring device 100 and the control device 200. In Figure 4B, only one of the multiple distance measuring devices 100 has a specific configuration. The other distance measuring devices 100 may have the same configuration. Note that the configuration may differ among the distance measuring devices 100. For example, some of the distance measuring devices 100 may be configured to output general distance data.

[0061] [Configuration of the rangefinder] The distance measuring device 100 shown in Figure 4B comprises a light source 110, a light receiving device 120, a processing circuit 130, and an input / output interface 150. In the example in Figure 4B, a clock 160 that outputs time data is provided outside the distance measuring device 100. The clock 160 outputs time data to multiple distance measuring devices 100. The clock 160 may also be provided inside the distance measuring device 100.

[0062] In this embodiment, the light source 110 is a means for outputting flash light that diffuses laser light over a wide area. The light source 110 includes, for example, a laser light source and a scattering plate, and emits light that spreads over a wide area by scattering the laser light with the scattering plate.

[0063] The light receiving device 120 includes an image sensor 121 and optical components (not shown). The optical components include, for example, one or more lenses that project light from a certain angle of view onto the light-receiving surface of the image sensor 121. The optical components may also include other optical elements such as prisms or mirrors. The optical components may be designed so that light diffused from one point of an object in the scene is focused onto one point on the light-receiving surface of the image sensor 121.

[0064] The image sensor 121 is a sensor in which multiple light-receiving elements 122 are arranged two-dimensionally along a light-receiving surface. The image sensor 121 comprises multiple light-receiving elements 122, multiple charge storage units 124, and multiple switches 123. Multiple (for example, three) charge storage units 124 are provided corresponding to each of the multiple light-receiving elements 122. A switch 123 is provided for each light-receiving element 122 and switches the connection between the light-receiving element 122 and the multiple charge storage units 124 corresponding to that light-receiving element 122. Each light-receiving element 122 receives light through photoelectric conversion. A charge corresponding to the amount of light is generated for each exposure period. Each charge storage unit 124 stores the charge generated by the photodetector 122 during the corresponding exposure period. The number of charge storage units 124 corresponding to each photodetector 122 is equal to or greater than the number of exposure periods required for the indirect ToF distance measurement operation. The switch 123 switches the connection between the photodetector 122 and the charge storage unit 124 according to the instructions from the processing circuit 130 and the switching of the exposure period. In the following description, a set consisting of one photodetector 122, a charge storage unit 124 corresponding to that photodetector 122, and a switch 123 corresponding to that photodetector 122 may be referred to as a "pixel".

[0065] The image sensor 121 may be, for example, a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The image sensor 121 may have detection sensitivity not only in the visible wavelength range, but also in wavelength ranges such as ultraviolet, near-infrared, mid-infrared, and far-infrared. The image sensor 121 may also be a sensor utilizing SPAD (Single Photon Avalanche Diode).

[0066] The image sensor 121 may, for example, be equipped with an electronic shutter system that exposes all pixels at once, i.e., a global shutter mechanism. The electronic shutter may be a rolling shutter system that exposes each row, or an area shutter system that exposes only a portion of the area corresponding to the irradiation range of the light beam. When the electronic shutter is a global shutter system, two-dimensional information can be acquired in one go by controlling the shutter in synchronization with the flash light. On the other hand, in the case of a system that changes the exposure timing for some pixels, such as a rolling shutter system, the amount of information that can be acquired decreases because pixels whose exposure timing does not match cannot receive light. However, this problem can be solved by performing a process to correct the shutter timing difference for each pixel. As shown in the modified examples described later, when the light source 110 is a beam scanner that emits a light beam with a small spread, the rolling shutter system may be more suitable than the global shutter system.

[0067] The light receiving device 120 receives light reflected from objects in the scene. For each frame, the light receiving device 120 outputs data indicating the charge accumulated for each exposure period by the indirect ToF described above for all pixels of the image sensor 121. In this embodiment, in the operation of one frame, the emission and exposure are repeated a common number of times for each of the multiple exposure periods so that sufficient charge is accumulated for distance calculation. When charge accumulation is complete for all exposure periods, the light receiving device 120 outputs data for all pixels for all exposure periods. The light receiving device 120 outputs the above data at a rate of, for example, 30 frames per second. The data output from the light receiving device 120 is recorded on a recording medium 170 such as memory.

[0068] The recording medium 170 may include memory such as ROM or RAM. The recording medium 170 records various types of data generated by the processing circuit 130. The recording medium 170 may also store computer programs executed by the processing circuit 130.

[0069] The processing circuit 130 determines the timing of the flash light emission from the light source 110 and the exposure timing of the light receiving device 120, and outputs an exposure control signal and a light emission control signal according to that timing. The processing circuit 130 also converts the charge accumulated in the charge storage unit 124 at each pixel of the light receiving device 120 for each exposure period into a pixel value for each exposure period, and outputs it as an array of pixel value data, i.e., image data representing the brightness of each pixel. This image data for each exposure period is sent to the control device 200 via the interface 150.

[0070] The processing circuit 130 is an electronic circuit that includes, for example, a processor such as a CPU. The processing circuit 130 performs the processing in this embodiment by executing, for example, a program stored in the recording medium 170. The recording medium 170 may be included in the processing circuit 130.

[0071] The dotted line frame 300 in Figure 4B schematically shows an example of image data for each exposure period sent from the interface 150 to the control device 200. In this example, exposure is performed for three exposure periods A0, A1, and A2 as shown in Figures 1 and 2. The output data from the distance measuring device 100 in this example includes a number of image data equal to the number of exposure periods per frame. In this embodiment, in addition to the image data, the output data includes time data that identifies each exposure period, which is necessary for the control device 200 to perform distance calculations, and data indicating the duration of the light pulses emitted from the light source 110. Details of the output data format will be described later.

[0072] The clock 160 is a circuit that outputs detailed time information necessary for controlling the light source 110. The clock 160 measures the time with, for example, nanosecond or microsecond precision and outputs the data. The clock 160 can be implemented by an integrated circuit, such as a real-time clock. The clock 160 may be synchronized with a time server. For synchronization, protocols such as NTP (Network Time Protocol) or PTP (Precision Time Protocol) may be used. Alternatively, time synchronization may be performed using GPS information based on the time of the control device 200. Note that the method of time synchronization is not limited to the above and is arbitrary. Through time synchronization, the distance measuring device 100 can acquire accurate time data.

[0073] [Control device configuration] Next, the configuration of the control device 200 in this embodiment will be described. As shown in Figure 4B, the control device 200 includes an input / output interface 210, a processing circuit 230, and a recording medium 270. The control device 200 controls the operation of the operating unit 240 in the vehicle. The operating unit 240 is a device that performs operations related to autonomous driving, such as the engine, steering, brakes, or accelerator.

[0074] Interface 210 acquires output data from multiple distance measuring devices 100. Interface 210 also acquires map data distributed from an external server (not shown). The map data may be, for example, landmark map data. Interface 210 may also be configured to acquire other data used by the processing circuit 230 for processing. The other data may be, for example, color image, tilt, velocity, or acceleration data acquired by the distance measuring devices 100 or other sensors.

[0075] The processing circuit 230 includes a pre-processing unit 231, a point cloud data generation unit 232, an environment recognition unit 233, and an operation control unit 234. The processing circuit 230 is an electronic circuit including a processor such as a CPU or GPU. The functions of the pre-processing unit 231, the point cloud data generation unit 232, the environment recognition unit 233, and the operation control unit 234 in the processing circuit 230 can be realized, for example, by the processor of the processing circuit 230 executing a program stored in the recording medium 270. In that case, the processor functions as the pre-processing unit 231, the point cloud data generation unit 232, the environment recognition unit 233, and the operation control unit 234. Each of these functional units may be realized by dedicated hardware. The recording medium 270 may be included in the processing circuit 230.

[0076] The preprocessing unit 231 performs image processing, such as noise reduction, on the output data from the distance measuring device 100 acquired via the interface 210 before performing distance calculations. This image processing will be referred to as preprocessing in the following description. Noise reduction processing is performed on image data for each exposure period. This process is performed on the data. Details of the noise reduction process will be described later. In addition to noise reduction, the preprocessing may also include processes such as edge extraction or smoothing.

[0077] The point cloud data generation unit 232 uses the images for each exposure period processed by the preprocessing unit 231 and the time data for each exposure period of each distance measuring device 100 to perform the aforementioned distance calculation for pixels at the same coordinates in each image. The point cloud data generation unit 232 calculates the distance for each pixel of the image sensor 121 of each distance measuring device 100, that is, for each position on the xy coordinate system, and generates distance image data. After generating distance image data for each distance measuring device 100, the point cloud data generation unit 232 converts the distance of each pixel in the distance image of each distance measuring device 100 into a point on the 3D coordinate system referenced by the control device 200, based on the position and direction data of each distance measuring device 100.

[0078] The processing circuit 230 extracts objects such as cars, people, and bicycles from the point cloud data in unified coordinates generated by the point cloud data generation unit 232. The processing circuit 230 further matches the point cloud data in unified coordinates with map data to recognize the state of its surroundings.

[0079] The motion control unit 234 determines the operation of the actuator 240 based on the position of the object in three-dimensional space identified by the environment recognition unit 233, and transmits a control signal to the actuator 240.

[0080] The operating unit 240 performs actions in accordance with control signals transmitted from the operation control unit 234. For example, it performs actions such as starting, accelerating, decelerating, stopping, and changing direction of the vehicle.

[0081] [Operation of the rangefinder] Next, the operation of the distance measuring device 100 will be explained.

[0082] Figure 5 is a flowchart showing the operation of the distance measuring device 100. In this embodiment, the processing circuit 130 of the distance measuring device 100 performs the operations from steps S1120 to S1190 shown in Figure 5. The operation of each step will be described below.

[0083] (Step S1120) The processing circuit 130 first determines whether or not an operation termination signal is input from an external device (not shown). If an operation termination signal is present in step S1120, the processing circuit 130 terminates its operation. If there is no operation termination signal in step S1120, the process proceeds to step S1130.

[0084] (Step S1130) The processing circuit 130 outputs a control signal to the light receiving device 120, and the light receiving device 120 opens the electronic shutter according to the control signal output by the processing circuit 130. This initiates the light detection operation for one frame.

[0085] (Step S1140) The processing circuit 130 determines whether all predetermined exposure periods necessary to generate distance data for one frame have finished. In this embodiment, a predetermined number of light projections and exposures are repeated during each exposure period. As a result, charge is accumulated in each pixel for all exposure periods. If charge accumulation for all exposure periods is completed in step S1140, the process proceeds to step S1170. If charge accumulation for all exposure periods is not completed in step S1140, the process proceeds to step S1150.

[0086] (Step S1150) The processing circuit 130 selects one of the predetermined exposure periods in which light emission and exposure have not yet been performed and charge accumulation has not occurred, and outputs a switching signal to the switch 123. This connects the light-receiving element 122 in each pixel to the charge storage unit 124 that stores the charge for that exposure period. Details of the switching timing will be described later.

[0087] (Step S1160) The processing circuit 130 refers to time data from the clock 160 and generates signals to control the light emission timing of the light source 110 and the exposure timing of the light receiving device 120 according to predetermined exposure start and end times for the selected exposure period. The light source 110 emits pulsed flash light of a predetermined duration according to the control signals output by the processing circuit 130. The light receiving device 120 performs exposure according to the predetermined start and end times of the exposure period, based on the start time of light emission from the light source 110, and stores the charge generated by photoelectric conversion during the exposure period in the charge storage unit 124 selected in step S1150. Details of the light emission and exposure timing will be described later.

[0088] Steps S1140 through S1160 are repeated to complete the light reception operation for one frame.

[0089] (Step S1170) If the processing circuit 130 determines in step S1140 that light emission and light reception have been completed for the entire exposure period, it closes the shutter of the light receiving device 120.

[0090] (Step S1180) The processing circuit 130 reads the charge accumulated in each charge storage unit 124 of each pixel during each exposure period through a series of operations from step S1130 to step S1160, converts the charge into a pixel value, and records it on the recording medium 170. The processing circuit 130 further erases the charge in each charge storage unit 124.

[0091] (Step S1190) The processing circuit 130 converts the charge for each exposure period read in step S1180 into pixel values ​​and generates luminance image data for each exposure period. Furthermore, in addition to the luminance image data for each exposure period, the processing circuit 130 adds timing data to identify each exposure period at predetermined intervals of a certain number of frames, or at the beginning of each frame. The timing data may include, for example, information on the start and end times of the exposure period. The processing circuit 130 outputs output data, which includes the luminance image data for each exposure period and the timing data to identify each exposure period, via the interface 150. Specific examples of the output data will be described later. After output, the process returns to step S1120.

[0092] The processing circuit 130 repeats the operations from step S1120 to step S1190. This unit of repetition is sometimes referred to as a "frame operation." By repeating multiple frame operations, data necessary for distance calculation is output for each frame. The output data is sent to the control device 200.

[0093] In this embodiment, in step S1190, output data is generated that includes luminance image data for each exposure period, as well as timing data for each exposure period. The output data may also include data on the light emission time, i.e., the duration of the light beam pulse.

[0094] [Examples of light projection and reception operations] Specific examples of the operation of the distance measuring device 100 in this embodiment, specifically the operation of light projection and exposure, will be described.

[0095] Figure 6 is a time chart showing an example of the operation of light projection and exposure by the distance measuring device 100. Figure 6 shows an example of the operation of light projection and exposure for one frame on the time axis. From top to bottom, it shows the light projection timing of the light source 110, the exposure timing of the light receiving device 120, the shutter open period of the light receiving device 120, and the readout timing of the charge storage unit 124. The shutter of the light receiving device 120 is opened in step S1130 when light projection and exposure for one frame begin and is not closed until the exposure operation is completed. The operation of one frame begins when the shutter is opened. In step S1150, one of several exposure periods is selected, and for each pixel, the charge storage unit 124 and the light receiving element 122 corresponding to that exposure period are connected by the switch 123.

[0096] In this embodiment, pulsed flashes of light from the light source 110 are repeatedly emitted during each of the exposure periods A0, A1, and A2. The pulse duration T0 may be, for example, about 90 nanoseconds (nsec). With the start of light emission as the starting point of time, the charge storage unit 124 and the light receiving element 122 corresponding to each exposure period are connected by the switch 123 from the start to the end of that exposure period. An example of the timing of two consecutive light emission and exposures in each exposure period is shown within the dotted line frame at the top of Figure 6.

[0097] During exposure period A0, the start and end times of exposure are the same as the start and end times of light emission. For example, the exposure start time is 0 nsec and the exposure end time is 90 nsec, which is the same as the duration of the emitted pulse. The switch is connected simultaneously with the light emission and disconnected simultaneously with the end of the light emission. Light emission by the light source 110 and light reception by the light receiving element 122 and charge accumulation in the charge storage unit 124 are repeated a predetermined number of times. Even if the energy of the reflected light is small in a single emission, it can be measured by charge accumulation through multiple light receptions.

[0098] When a predetermined number of light projections and exposures are completed during exposure period A0, the next exposure period A1 is selected. In the newly selected exposure period A1, the charge storage unit 124 and the photodetector 122 corresponding to that exposure period are connected by the switch 123. The timing of the flash light projection by the light source 110 does not change even when the exposure period changes. On the other hand, in the newly selected exposure period A1, as shown in the enlarged view of Figure 6, exposure starts when a pulse duration T0 (for example, 90 nsec) has elapsed from the start of light projection, and exposure ends when a duration T0 has elapsed after the start of exposure. That is, the switch 123 connects the charge storage unit 124 and the photodetector 122 corresponding to that exposure period so that exposure starts simultaneously with the end of light projection and exposure lasts for the same duration T0 as the pulse duration of light projection. Similar to exposure period A0, the charge storage unit 124 holds the charge accumulated by repeated exposures.

[0099] Once a predetermined number of light projections and exposures are completed in exposure period A1, light projections and exposures are performed again in exposure period A2. In exposure period A2, as shown in the enlarged view of Figure 6, exposure begins 2T0 (for example, 180 nsec) after the start of light projection, and ends T0 after the start of exposure. That is, exposure begins again after a further light projection duration T0 has elapsed after the end of light projection. The exposure duration in exposure period A2 is also T0, similar to exposure periods A0 and A1. In exposure period A2, charge is accumulated in the charge accumulation unit 124 by repeating the predetermined number of light projections and exposures.

[0100] Once charge accumulation is complete for all exposure periods A0, A1, and A2, the processing circuit 130 causes the light-receiving device 120 to close its shutter (step S1170). The processing circuit 130 then reads out the charge accumulated in the charge accumulation unit 124 of each pixel corresponding to each exposure period. The processing circuit 130 generates and outputs brightness image data for each exposure period based on the read charge.

[0101] [Example of data format] Next, an example of the data format output by the distance measuring device 100 of this embodiment will be described.

[0102] Figures 7A and 7B show examples of the output format of image data for each exposure period output from the interface 150 of the distance measuring device 100. In this example, the output data includes fixed value data common to multiple frames and data that differs for each frame. As data common to multiple frames, for example, a fixed value is output once at the beginning of the output data, or once for a predetermined number of frames.

[0103] The fixed values ​​include, for example, data for position, direction, field of view, pixel arrangement, exposure period A0, exposure period A1, and exposure period A2. "Position" indicates the position of the image sensor 121 within the vehicle. The position may be, for example, 3 bytes of data expressed in 3D coordinates with the center of the vehicle as the origin. "Direction" indicates the direction that the light-receiving surface of the image sensor 121 faces. The direction may be, for example, 3 bytes of data representing the normal vector of the light-receiving surface expressed in 3D coordinates with the center of the vehicle as the origin. "Field of view" indicates the field of view of the image sensor 121 and may be expressed in, for example, 2 bytes. "Pixel arrangement" indicates the number of pixels in the x and y directions of the image sensor 121 and may be, for example, 1 byte of data. "Exposure period A0," "Exposure period A1," and "Exposure period A2" indicate the time range of each exposure period. These time ranges may be expressed in, for example, 1 byte of data describing the elapsed time from the start of the corresponding light projection in nanoseconds. Note that the number of exposure periods per frame may be a number other than 3. Furthermore, the number of exposure periods may change during operation. For example, the number of exposure periods may be defined separately as a fixed value, and time data for each exposure period may be output.

[0104] The output data for each frame includes, for example, the date, time, luminance image data for exposure period A0, luminance image data for exposure period A1, and luminance image data for exposure period A2. The date is, for example, data indicating the year, month, and day, and can be represented by 1 byte. The time is, for example, data indicating the hour, minute, second, millisecond, and microsecond, and can be represented by 5 bytes. The luminance image for exposure period A0 is a set of pixel values ​​converted from the charge of each pixel accumulated during exposure period A0, and can be represented, for example, as 1 byte of data per pixel. Similarly, the luminance images for exposure period A1 and exposure period A2 can also be represented, for example, as 1 byte of data per pixel.

[0105] [Control device operation] Next, an example of the operation of the control device 200 will be described.

[0106] Figure 8 is a flowchart showing the operation of the control device 200 in this embodiment. The processing circuit 230 of the control device 200 executes the operations from steps S2120 to S2200 shown in Figure 8. The operation of each step will be described below.

[0107] (Step S2120) The processing circuit 230 determines whether or not an operation termination signal has been input from an external device (not shown). If an operation termination signal has been input, the operation is terminated. If no operation termination signal has been input, the process proceeds to step S2130.

[0108] (Step S2130) The processing circuit 230 determines whether or not there is data input from the distance measuring device 100. If there is data input from the distance measuring device 100, the process proceeds to step S2140. If no data is entered, return to step S2120.

[0109] (Step S2140) The processing circuit 230 performs preprocessing on the data acquired from the distance measuring device 100 to improve the accuracy of distance calculation. Preprocessing includes, for example, noise reduction. Here, the distance measuring device 100 outputs data in the format shown in Figures 7A and 7B. In this case, preprocessing is performed on the image data corresponding to exposure period A0, the image data corresponding to exposure period A1, and the image data corresponding to exposure period A2. The processing circuit 230 performs noise reduction processing individually on the image data corresponding to each exposure period. As a method of noise reduction processing, for example, adaptive filtering using a Wiener filter may be performed.

[0110] The processing circuit 230 removes noise by filtering out high spatial frequency components in each image data. The filter may be a filter other than a Wiener filter, such as a Gaussian filter. To remove high spatial frequency components in each image data, smoothing may be performed by convolution using a predetermined filter, such as a Laplacian filter. As a preprocessing step, noise reduction processing other than adaptive filtering may be performed. In addition, signal processing other than noise reduction processing, such as contrast enhancement or edge extraction, may be performed.

[0111] (Step S2150) The processing circuit 230 calculates the distance for each pixel using the image data for each exposure period that has undergone noise reduction processing in step S2140. The processing circuit 230 extracts the pixel values ​​of the same pixels from the image data for each exposure period, calculates the flight time based on the calculation formulas (4) and (5) described above, and then calculates the distance.

[0112] (Step S2160) The processing circuit 230 further calculates the reflectance of the pixels whose distances have been calculated, using the distances to each pixel calculated in step S2150 and the image data for each exposure period that has undergone noise reduction processing in step S2140. The reflectance is the ratio of the sum of the pixel values ​​for each exposure period, with background noise removed, to the value at 100% reflectance, which is predetermined for each distance. The value at 100% reflectance can be obtained, for example, by pre-measuring the pixel values ​​of the reflected light from a reference white board at each distance from the light receiving device 120. The 100% reflectance values ​​for each distance can be pre-recorded on the recording medium 270 in the form of a table or the like.

[0113] Figure 9 shows an example of a table recorded on the recording medium 270. In this example, the table defines the correspondence between the distance of the light-receiving device 120 from the light-receiving surface and the pixel value when reflected light from a hypothetical white board with 100% reflectivity located at that distance is detected. The value of 100% reflectivity may be recorded as a function of distance. The table or function defining the correspondence between distance and the value of 100% reflectivity may be transmitted from the distance measuring device 100 to the control device 200.

[0114] (Step S2170) The processing circuit 230 converts the distance data for each pixel calculated in step S2150, i.e., distance image data, into 3D point cloud data. Here, pixels in the distance image data whose distance value is 0 or infinite are not considered points in the point cloud data and are not converted. Only pixels in the distance image data for which a valid distance has been calculated are converted. The conversion is performed, for example, as follows: First, the position data of the distance measuring device 100 shown in Figures 7A and 7B is used to determine the coordinate system of the distance measuring device 100. The distance between the origin and the origin of the coordinate system for data integration set in the control device 200 is calculated. Furthermore, the amount of rotation of the coordinate axes of the distance measuring device 100 relative to the coordinate axes of the integrated coordinate system of the control device 200 is calculated by referring to the direction data of the distance measuring device 100 shown in Figures 7A and 7B. Based on the calculated distance and rotation amount, a coordinate transformation is performed on the pixels in the distance image generated in step S2150 that have valid distance values.

[0115] (Step S2180) The processing circuit 230 recognizes the environment around the distance measuring device 100 based on the point cloud data generated in step S2170 and map data acquired from an external source. Environment recognition can be performed by matching the map data with the point cloud data. For example, the processing circuit 230 identifies fixed objects from the point cloud data that match fixed objects included in the map, and determines the positional relationship between those fixed objects and the distance measuring device 100. Furthermore, the processing circuit 230 groups the point cloud data by object using the reflectivity information for each point calculated in step S2160. This makes it possible to extract moving objects around the distance measuring device 100 that are not included in the map, such as people, animals, bicycles, or automobiles.

[0116] Matching map data and point cloud data can be performed, for example, as follows: First, a landmark map is referenced, a landmark to be detected is selected, and information such as the location coordinates of that landmark is read. Then, the distance to the landmark is calculated based on the output data of the distance measuring device 100. Matching can be performed based on the coordinates of the landmark and the calculated distance to the landmark.

[0117] Since the pre-processed luminance image data and point cloud data are generated based on data output from the same distance measuring device 100, there is no pose shift. Therefore, the image data and point cloud data can be superimposed without distortion correction. The processing circuit 230 may identify objects by extracting objects that overlap with objects obtained by point cloud grouping from the luminance image data. Even when it is difficult to clarify the boundaries of objects using only point cloud grouping, the boundaries of objects become clearer by comparing them with the results of pixel grouping using luminance images without positional shift. In this way, by integrating information obtained from both luminance image data and point cloud data, it becomes possible to reproduce the scene more accurately as a three-dimensional space. When extracting objects from luminance image data, well-known image recognition processes such as AI-based object recognition can be used. In addition, image processing such as edge extraction and contrast enhancement may be performed as pre-processing for the luminance image data.

[0118] (Step S2190) The processing circuit 230 determines the operation of the operating parts 240, such as brakes, accelerators, and steering wheels, based on the arrangement of structures such as buildings and moving objects in the three-dimensional space extracted in step S2180.

[0119] (Step S2200) The processing circuit 230 generates and outputs a control signal to control the operating unit 240 based on the operation of the operating unit 240 determined in step S2190.

[0120] [effect] As described above, the distance measuring device 100 of this embodiment generates luminance image data for each of the multiple exposure periods by performing light projection and exposure during each of the multiple exposure periods. The control device 200 calculates the distance of each pixel based on the luminance image data for each exposure period and further converts the distance data into 3D point cloud data. Before calculating the distance, the control device 200 performs preprocessing such as noise reduction on the luminance image data for each exposure period. This makes the distance more accurate. This can be calculated. Furthermore, by calculating the reflectance lost during distance calculation, the accuracy of object extraction or recognition based on point cloud data can be improved.

[0121] The configuration of this embodiment is merely an example, and various modifications are conceivable. Several modifications of this embodiment are described below.

[0122] <Modification 1 of Embodiment 1> In Embodiment 1, the light source 110 emits a flash of light that diffuses the laser beam over a wide area, but the light source 110 may also be configured to emit a light beam with a smaller spread than the flash of light. By using a light beam such as laser light, the energy density of the light can be increased compared to when using a flash of light. Therefore, reflected light from objects at a greater distance can be detected. In order to project light over a wide area, the light source 110 is controlled to project light multiple times by changing the direction of the light.

[0123] [Configuration of the rangefinder] The configuration of the distance measuring device 100 in this modified example is the same as that shown in Figure 4B. However, the function and operation of the light source 110 are different. The configuration and operation of this modified example will be described below, focusing on the differences from Embodiment 1.

[0124] In this modified example, the light source 110 is a beam scanner capable of changing the direction of emission of the light beam. In response to commands from the processing circuit 130, the light source 110 sequentially illuminates a portion of the scene with the light beam. To achieve this function, the light source 110 is equipped with a mechanism for changing the direction of emission of the light beam.

[0125] Figure 10 shows an example of a light source 110. In this example, the light source 110 comprises a light-emitting element such as a laser and at least one movable mirror, such as a MEMS mirror. Light emitted from the light-emitting element is reflected by the movable mirror and directed towards a predetermined area in the scene. The processing circuit 130 can change the direction of the light beam's emission by driving the movable mirror. This allows, for example, the scene to be scanned with the light beam in one or two dimensions.

[0126] A light source capable of changing the direction of light emission by a structure other than one having a movable mirror may be used. For example, a light source utilizing a reflective waveguide, such as the one disclosed in Patent Document 4, may be used.

[0127] Figure 11A is a schematic perspective view showing an example of a light source 110 utilizing a reflective waveguide. For reference, mutually orthogonal X, Y, and Z axes are schematically shown. The light source 110 comprises an optical waveguide array 10A, a phase shifter array 20A, an optical splitter 30, and a substrate 40 on which they are integrated. The optical waveguide array 10A includes a plurality of optical waveguide elements 10 arranged in the Y direction. Each optical waveguide element 10 extends in the X direction. The phase shifter array 20A includes a plurality of phase shifters 20 arranged in the Y direction. Each phase shifter 20 has an optical waveguide extending in the X direction. The plurality of optical waveguide elements 10 in the optical waveguide array 10A are connected to the plurality of phase shifters 20 in the phase shifter array 20A, respectively. The optical splitter 30 is connected to the phase shifter array 20A.

[0128] Light L0 emitted from a light-emitting element (not shown) is input to multiple phase shifters 20 in the phase shifter array 20A via an optical splitter 30. The light that has passed through the multiple phase shifters 20 is input to each of the multiple optical waveguide elements 10 with its phase shifted by a fixed amount in the Y direction. The light input to each of the multiple optical waveguide elements 10 is emitted as an optical beam L2 from an optical emission surface 10s parallel to the XY plane in a direction intersecting the optical emission surface 10s.

[0129] Figure 11B is a schematic diagram showing an example of the structure of an optical waveguide element 10. The optical waveguide element 10 includes an optical waveguide layer 15 located between a first mirror 11 and a second mirror 12 facing each other, and a pair of electrodes 13 and 14 for applying a driving voltage to the optical waveguide layer 15. The optical waveguide layer 15 may be made of a material whose refractive index changes with the application of voltage, such as a liquid crystal material or an electro-optic material. The transmittance of the first mirror 11 is higher than that of the second mirror 12. Each of the first mirror 11 and the second mirror 12 may be formed from, for example, a multilayer reflective film in which a plurality of high refractive index layers and a plurality of low refractive index layers are alternately stacked.

[0130] Light input to the optical waveguide layer 15 propagates along the X direction, being reflected by the first mirror 11 and the second mirror 12 within the optical waveguide layer 15. The arrows in Figure 11B schematically represent the propagation of light. A portion of the light propagating within the optical waveguide layer 15 is emitted to the outside through the first mirror 11.

[0131] By applying a driving voltage to electrodes 13 and 14, the refractive index of the optical waveguide layer 15 changes, and the direction of the light emitted from the optical waveguide element 10 to the outside changes. In response to the change in driving voltage, the direction of the optical beam L2 emitted from the optical waveguide array 10A changes. Specifically, the emission direction of the optical beam L2 shown in Figure 11A can be changed along a first direction D1 parallel to the X-axis.

[0132] Figure 11C is a schematic diagram illustrating an example of a phase shifter 20. The phase shifter 20 includes, for example, a total reflection waveguide 21 containing a thermo-optical material whose refractive index changes with heat, a heater 22 that is thermally in contact with the total reflection waveguide 21, and a pair of electrodes 23 and 24 for applying a driving voltage to the heater 22. The refractive index of the total reflection waveguide 21 is higher than that of the heater 22, the substrate 40, and air. Due to the refractive index difference, light input to the total reflection waveguide 21 propagates along the X direction while being totally reflected within the waveguide 21.

[0133] By applying a driving voltage to the pair of electrodes 23 and 24, the total reflection waveguide 21 is heated by the heater 22. As a result, the refractive index of the total reflection waveguide 21 changes, and the phase of the light output from the end of the total reflection waveguide 21 shifts. By changing the phase difference of the light output from two adjacent phase shifters 20 in the plurality of phase shifters 20 shown in Figure 11A, the emission direction of the light beam L2 can be changed along a second direction D2 parallel to the Y axis.

[0134] With the above configuration, the light source 110 can change the emission direction of the light beam L2 in two dimensions.

[0135] Details of the operating principle and operating method of the light source 110 described above are disclosed, for example, in Patent Document 4. The entirety of the disclosure in Patent Document 4 is incorporated herein by reference.

[0136] In this modified example, the processing circuit 130 records the charge accumulated when a light-receiving element in a certain area of ​​the image sensor 121 receives light, in association with the time of the light beam's emission, on the recording medium 170 for each light beam emission.

[0137] Figure 12 shows an example of data that can be recorded on the recording medium 170. The recording medium 170 stores a representative value of the time of light emission and a time ID representing the representative value of the light emission time, associated with each light beam emission a predetermined number of times. In addition, the charge accumulated by the charge storage unit 124 in conjunction with the emission of the light beam and the time ID corresponding to that emission are stored for each photodetector 122, i.e., for each pixel. Since the light beam is emitted with each exposure, the charge and time ID are recorded with each exposure.

[0138] [Operation of the rangefinder] Next, the operation of the distance measuring device 100 in this modified example will be explained. Figure 13 is a flowchart of the operation of the distance measuring device 100 in this modified example. In the flowchart shown in Figure 13, steps S3110 and S3120 are added to the flowchart shown in Figure 5, and step S1180 is changed to step S3130. The operation of each step will be explained below.

[0139] (Step S1120) The processing circuit 130 first determines whether or not an operation termination signal is input from an external device (not shown). If an operation termination signal is present in step S1120, the processing circuit 130 terminates its operation. If there is no operation termination signal in step S1120, the process proceeds to step S3110.

[0140] (Step S3110) The processing circuit 130 determines whether the operation has been completed for all of the one or more light projection directions that are predetermined or specified by an external device not shown. If it is determined that the operation has been completed for all light projection directions, the process proceeds to step S1190. If it is determined that the operation has not been completed for all light projection directions, the process proceeds to step S3120.

[0141] (Step S3120) The processing circuit 130 selects one of the one or more light projection directions that have not yet been projected, from among those predetermined or specified by an external device not shown.

[0142] (Step S1130) The processing circuit 130 outputs a control signal to the light receiving device 120, and the light receiving device 120 opens the electronic shutter according to the control signal output by the processing circuit 130. This initiates the light detection operation for one light emission direction.

[0143] (Step S1140) The processing circuit 130 determines whether all predetermined exposure periods necessary for generating distance data have been completed. If charge accumulation for all exposure periods has been completed in step S1140, the process proceeds to step S1170. If charge accumulation for all exposure periods has not been completed in step S1140, the process proceeds to step S1150.

[0144] (Step S1150) The processing circuit 130 selects one of the predetermined exposure periods in which light emission and exposure have not yet been performed and charge accumulation has not occurred, and outputs a switching signal to the switch 123.

[0145] (Step S1160) The processing circuit 130 refers to time data from the clock 160 and generates signals to control the light emission timing of the light source 110 and the exposure timing of the light receiving device 120 according to predetermined exposure start and end times for the selected exposure period. The light source 110 emits a pulsed light beam of a predetermined duration in the direction determined in step S3120, according to the control signals output by the processing circuit 130. The light receiving device 120 performs exposure according to the predetermined start and end times of the exposure period, with reference to the start of light emission from the light source 110, and processes the charge generated by photoelectric conversion during the exposure period in step S The charge is stored in the charge storage unit 124 selected at 1150.

[0146] (Step S1170) If the processing circuit 130 determines in step S1140 that light emission and light reception have been completed for the entire exposure period, it closes the shutter of the light receiving device 120.

[0147] (Step S3130) The processing circuit 130 reads out the charge accumulated during each exposure period through the series of operations from step S1130 to step S1160, converts the charge of the pixels that received reflected light into a pixel value, and records the pixel value for each pixel on the recording medium 170.

[0148] (Step S1190) The processing circuit 130 generates luminance image data for each exposure period based on the data recorded in step S3130. Furthermore, in addition to the luminance image data for each exposure period, the processing circuit 130 generates data on the light beam emission time. For example, in addition to the luminance image data for each exposure period, the processing circuit 130 adds timing data to identify each exposure period at predetermined intervals or at the beginning of each frame. The timing data may include, for example, information on the start and end times of the exposure period. The processing circuit 130 outputs output data, which includes the luminance image data for each exposure period, the timing data to identify each exposure period, and the time data for each pixel, via the interface 150. After output, the process returns to step S1120.

[0149] The processing circuit 130 repeats the operations from step S1120 to step S1190. This unit of repetition is called a "frame operation." By repeating multiple frame operations, data necessary for distance calculation is output for each frame. The output data is sent to the control device 200.

[0150] [Example of data format] Figures 14A and 14B show examples of the output format of image data for each exposure period output from the distance measuring device 100 in this modified example. In this example, compared to the examples shown in Figures 7A and 7B, the number of light projection directions (e.g., 1 byte) is added to the fixed value. In addition, the data for each frame includes time data (e.g., 5 bytes) for each light projection direction for each exposure period. The luminance value data for each exposure period is appended with time ID data (e.g., 1 byte) that identifies a representative value of the emission time of the light beam used to acquire that luminance value. The correspondence between the time ID and the actual time is recorded on the recording medium 170, as shown in Figure 12. The data defining the correspondence between the time ID and the actual time is also pre-recorded on the recording medium 270 of the control device 200.

[0151] Figures 15A and 15B show other examples of the output format of image data for each exposure period output from the distance measuring device 100. In this example, a set of pixels that receive reflected light from a light beam emitted in the same direction is treated as one block, and data (e.g., 4 bytes) that identifies the pixel area of ​​each block is included in the output data for each frame. Each block is, for example, a rectangular pixel area and can be identified by the coordinates of the top-leftmost pixel and the bottom-rightmost pixel. The block data may also include a block ID. In the example in Figure 15B, after the date data, the pixel area data of the block is listed for the number of projection directions, followed by time data for each projection direction, listed for the product of the number of directions and the number of exposure periods. This is followed by image data obtained by converting the charge of each pixel accumulated in exposure period A0 into a pixel value, image data corresponding to the charge accumulated in exposure period A1, and image data corresponding to the charge accumulated in exposure period A2. Note that when outputting block information as in this example, data identifying the pixel area of ​​the block can be recorded on the recording medium 170. In this case, data such as luminance values ​​may be recorded on the recording medium 170 in block units. In the data formats shown in Figures 14B, 15A, and 15B, the time at which the luminance data was acquired is assigned to each pixel or pixel block. That is, luminance image data for one frame contains luminance data at different times. Luminance image data, distance image data, and point cloud data generated from such data formats can be used not on a frame-by-frame basis, but by dividing them into pixels or pixel blocks included in a predetermined time range. For example, when the control device 200 combines data acquired from the distance measuring device 100 with data including time information acquired from other measuring devices, it can divide and combine the data into pixels or pixel blocks based on the time information.

[0152] <Modification 2 of Embodiment 1> Next, a modification 2 of Embodiment 1 will be described. In this modification, unlike the examples described above, the light source 110 is configured to emit a laser beam with a small spread multiple times, changing its direction in one dimension. Unlike the examples described above, the light receiving device 120 is not an image sensor, but comprises one or a few light-receiving elements. The orientation of the light receiving device 120 is controlled in accordance with the change in the light-receiving direction of the light source 110, so that the normal vector of the light-receiving surface of the light receiving device 120 matches the light-emitting direction of the light source 110. Therefore, the processing circuit 130 records the data acquired by light reception on the recording medium 170 not for each light-receiving element (pixel), but for each light-emitting direction of the beam. The light-emitting direction of the beam is specified, for example, by an angle from a reference position.

[0153] The operation of the distance measuring device 100 in this modified example is the same as that of the distance measuring device 100 in the previously described modified example 1, so a description is omitted. For each light projection, the light receiving device 120 outputs luminance data corresponding to the charge accumulated during each exposure period as a data sequence for each frame. In this modified example, one frame operation refers to a set of operations of projection and exposure in multiple directions around a predetermined axis. In one frame operation, for example, a light beam can be sequentially emitted in a total of 720 directions by changing the direction by 0.5 degrees in a 360-degree horizontal range.

[0154] Figures 16A and 16B show examples of the output format of the data output from the distance measuring device 100 in this modified example. In this modified example, the fixed value is, instead of the field of view and pixel arrangement data in the example shown in Figures 14A and 14B, the range of angles in the direction from which the light beam is emitted in one frame operation, which is output in, for example, 2 bytes. In this modified example, "direction" is data that represents the 0-degree direction, which is pre-set in the distance measuring device 100, as a vector in a 3D coordinate table with the center of the vehicle as the origin.

[0155] In this modified version, for each frame's output data, first, the date (e.g., 1 byte) when the data for that frame was acquired is output. Subsequently, the brightness value (e.g., 1 byte) and time (e.g., 5 bytes) corresponding to the charge accumulated during exposure period A0 are repeatedly output for all projection directions. Similarly, for exposure periods A1 and A2, the brightness value and time corresponding to the accumulated charge are repeatedly output for all projection directions.

[0156] Figures 17A and 17B show other examples of the output format of the data output from the distance measuring device 100. In this example, for each frame, the output data first outputs the date (e.g., 1 byte), followed by the time for each projection direction during each exposure period (e.g., 5 bytes). Subsequently, the brightness value (e.g., 1 byte) corresponding to the charge accumulated during exposure period A0 is repeatedly output for all projection directions. Similarly, for exposure periods A1 and A2, the brightness value corresponding to the accumulated charge is repeatedly output for all projection directions.

[0157] <Embodiment 2> Next, an exemplary embodiment 2 of the present disclosure will be described.

[0158] In Embodiment 1 and its various modifications, the distance measuring device 100 always outputs luminance data indicating the charge accumulated by light reception during each exposure period, without calculating distance. In contrast, in this embodiment, the distance measuring device 100 calculates the distance for each pixel based on the luminance data and can switch between outputting distance data and outputting luminance data depending on the situation. This output switching can be based on external instructions or on internal data-based judgments. After receiving data from the distance measuring device 100, the control device 200 determines whether it is luminance data or distance data and performs processing according to the type of data.

[0159] Figure 18 shows the functional configuration of the system in this embodiment. The hardware configuration in this embodiment is the same as that shown in Figure 4B, but the operation differs from that of Embodiment 1. The configuration and operation of this embodiment will be described below, focusing on the differences from Embodiment 1. In the following description, luminance data may be referred to as raw data.

[0160] [Configuration of the rangefinder] In this embodiment, the processing circuit 130 has the function of calculating the distance for each pixel based on the raw data. The processing circuit 130 switches between outputting distance data and outputting raw data in response to a request from an external control device 200. In the system shown in Figure 18, there is one distance measuring device 100, but the system may include multiple distance measuring devices 100, similar to the example shown in Figure 4B.

[0161] The light source 110 in this embodiment, like the light source 110 in Embodiment 1, outputs a flash light that diffuses the laser light over a wide area.

[0162] The light-receiving device 120 includes an image sensor 121 and optical components such as lenses. The image sensor 121 has a plurality of pixels arranged in two dimensions. Each pixel includes a light-receiving element 122, a plurality of charge storage units 124 corresponding to a plurality of exposure periods, and a switch 123 for switching the connection between the light-receiving element 122 and the respective charge storage unit 124.

[0163] The processing circuit 130 determines the timing of the flash light emission by the light source 110 and the exposure timing of the light receiving device 120. Then, according to the determined timing, it sends a light emission control signal to the light source 110 and an exposure control signal to the image sensor 121. The processing circuit 130 further calculates the distance for each pixel based on the signals generated by the charge storage unit 124 for each exposure period. The processing circuit 130 outputs distance data, i.e., distance image data, indicating the calculated distance for each pixel, via the interface 150. On the other hand, when the processing circuit 130 receives a signal from the control device 200 requesting the output of raw data, it outputs raw data instead of distance data.

[0164] [Control device configuration] The control device 200 has the same hardware configuration as the control device 200 shown in Figure 4B. However, this embodiment differs from Embodiment 1 in that the processing circuit 230 sends a signal to the distance measuring device 100 via the interface 210 to request the transmission of raw data, depending on the situation. The processing circuit 230 sends a signal to the distance measuring device 100 requesting raw data when more accurate data is needed, for example, when the distance image data transmitted from the distance measuring device 100 contains a lot of noise.

[0165] In the configuration shown in Figure 18, signals are transmitted and received directly between the interface 150 of the distance measuring device 100 and the interface 210 of the control device 200. However, the configuration is not limited to this; communication between the distance measuring device 100 and the control device 200 may also be conducted via a network such as the Internet. A device may be interposed. The distance measuring device 100 or the control device 200 may communicate with a storage device such as a cloud server or storage via a network. Communication protocols such as http, ftp, TCP or UDP, and IP may be used for communication. A pull-type communication method or a push-type communication method may be used. For wired transmission, for example, Ethernet, USB, RS-232C, HDMI®, or coaxial cable may be used. Alternatively, for wireless transmission, any wireless communication method may be used, for example, 3GPP, 3G / 4G / 5G as defined by IEEE, wireless LAN, Wi-Fi, Bluetooth®, or millimeter wave.

[0166] [Operation of the rangefinder] Figure 19 is a flowchart showing the operation of the processing circuit 130 of the distance measuring device 100 in this embodiment. The flowchart shown in Figure 19 is the same as the flowchart shown in Figure 5, but with step S1190 replaced by steps S4110 to S4150. The operation from steps S1120 to S1170 is the same as the operation shown in Figure 5. The differences from the operation shown in Figure 5 will be explained below.

[0167] (Step S4110) Once step S1180 is complete, the processing circuit 130 determines whether or not it has received a signal from the control device 200 requesting raw data output. If the signal has been received, the process proceeds to step S4140. If the signal has not been received, the process proceeds to step S4120.

[0168] (Step S4120) The processing circuit 130 calculates the distance for each pixel based on the charge value of each pixel for each exposure period recorded on the recording medium 170. The method for calculating the distance is the same as the calculation method in step S2150 shown in Figure 8.

[0169] (Step S4130) The processing circuit 130 converts the distance for each pixel calculated in step S4120 into pixel values ​​to generate distance image data. The processing circuit 130 then generates output data by adding an identifier to the distance image data that indicates that the data is distance image data. A specific example of the output data will be described later.

[0170] (Step S4140) If the processing circuit 130 determines in step S4110 that there is a request for output of raw data, it converts the charge values ​​for each exposure period recorded on the recording medium 170 into pixel values ​​and generates luminance image data for each exposure period. In this case, the processing circuit 130 generates output data by adding timing data indicating the timing of each exposure period to the luminance image data for each exposure period. The format of the output data is the same as, for example, the format shown in Figures 7A and 7B. However, an identifier indicating that the data is raw data, i.e., luminance image data, is added to the beginning of the output data.

[0171] (Step S4150) The processing circuit 130 outputs the output data generated in step S4130 or step S4140 via the interface 150.

[0172] The processing circuit 130 repeatedly performs the operations from step S1120 to step S4150. This switches the output of distance image data and the output of luminance image data for each exposure period in response to a request from the external control device 200. The switching between the output of distance image data and the output of luminance image data for each exposure period can be performed at any frame. Alternatively, the processing circuit 130 may switch the output format every one or more predetermined number of frames. In this case, regardless of the request from the control device 200, the processing circuit 130 switches the output of distance image data and the output of luminance image data every predetermined number of frames.

[0173] [Example of data format] Next, an example of the data format output by the distance measuring device 100 in this embodiment will be described.

[0174] Figures 20A and 20B show examples of the format of distance image data output from the interface 150 of the distance measuring device 100. In this example, a fixed value common to multiple frames is preceded by an identifier indicating the data format, i.e., whether it contains distance data or luminance data. If the data format is distance data, the fixed value does not include the data for exposure periods A0 to A2 shown in Figure 7B. The fixed value can be output, for example, once at the beginning of the output data, or before the first frame when switching output data, as in the examples described above.

[0175] The output data for each frame includes the date (e.g., 1 byte) and time (e.g., 5 bytes) when the data for that frame was generated, and the value (e.g., 1 byte) obtained by converting the distance of each pixel into a pixel value. A distance image is constructed from the pixel value data of multiple pixels.

[0176] Figures 21A and 21B show examples of the format of luminance image data output from the distance measuring device 100. In these examples, the format of the luminance image data is the same as in the examples shown in Figures 7A and 7B, with an identifier (e.g., 1 byte) added to the beginning of the fixed values ​​to indicate that the data format is luminance image data.

[0177] Figures 22A and 22B show other examples of the format of luminance image data output from the distance measuring device 100. In this example, the format shown in Figures 22A and 22B is supplemented with a fixed value representing the maximum pixel value (e.g., 1 byte) pre-measured for each distance. This maximum pixel value represents the sum of pixel values ​​measured at one pixel during exposure periods A0 to A2, assuming that light is reflected from an object with 100% reflectivity (e.g., a white board) for each of several pre-set distances. This maximum pixel value can be recorded for multiple distances within a specific distance range (e.g., from 0.1m to 50m), for example, with the increment increasing as the distance increases. The increment can be determined, for example, in proportion to the logarithm of the distance with base 2. The data for the maximum pixel value is pre-measured for each distance and recorded in a table or other format on the recording medium 270, for example, as shown in Figure 9. The data for the maximum pixel value can be used, as described above, when the processing circuit 230 in the control device 200 calculates the reflectivity for each pixel. In Embodiment 1 and its various modifications, the output data may also include data representing the maximum value of such pixel values.

[0178] [Control device operation] Figure 23 is a flowchart showing an example of the operation of the processing circuit 230 in the control device 200 in this embodiment. In the operation shown in Figure 23, step S5110 is added between steps S2130 and S2140 in the flowchart shown in Figure 8, and steps S5120 to S5140 are added between steps S2170 and S2180. In all other respects, it is the same as the operation shown in Figure 8. The differences from the operation shown in Figure 8 will be mainly explained below.

[0179] (Step S5110) When the processing circuit 230 receives data from the distance measuring device 100 in step S2130, it determines whether the data is raw data, i.e., luminance image data. This determination is made based on the data format value of the input data at a fixed value. If the input data is raw data, the process proceeds to step S2140. If the input data is not raw data, i.e., distance image data, the process proceeds to step S2170.

[0180] (Steps S2140~S2160) The processing in steps S2140, S2150, and S2160 is the same as the processing in the corresponding steps in Figure 8. In step S2140, the processing circuit 230 performs preprocessing on the raw data, such as noise reduction, to improve the accuracy of distance calculation. In step S2150, the processing circuit 230 generates distance image data by calculating the distance for each pixel using the image data for each exposure period that has undergone noise reduction in step S2140. In step S2160, the processing circuit 230 calculates the reflectance of the pixels for which distance has been calculated, using the distance for each pixel calculated in step S2150 and the image data for each exposure period that has undergone noise reduction in step S2140. The 100% reflectance value for each distance used in the calculation of reflectance may be pre-recorded on the recording medium 170, as in Embodiment 1, or it may be included as a fixed value in the input data, as shown in Figures 22A and 22B.

[0181] (Step S2170) The processing circuit 230 converts the distance image data generated in step S2150, or the distance image data input from the distance measuring device 100, into point cloud data. This conversion process is the same as the process in step S2170 in Figure 8.

[0182] (Step S5120) The processing circuit 230 estimates its own position based on the point cloud data generated in step S2170 and map data acquired from an external source. The self-position can be estimated, for example, by matching the map data with the point cloud data. For example, the processing circuit 230 identifies fixed objects that match fixed objects included in the map from the point cloud data through matching. Then, the self-position can be estimated based on the distance value to the fixed object and the distance to the fixed object obtained from the map.

[0183] (Step S5130) The processing circuit 230 determines whether the position of the control device 200, estimated in step S5120, is within a region that meets predetermined conditions. These conditions may include, for example, being within 10 meters of an intersection. If the position of the control device 200 meets the conditions, the process proceeds to step S5140. If the position of the control device 200 does not meet the conditions, the process proceeds to step S2190.

[0184] (Step S5140) The processing circuit 230 transmits a signal to the distance measuring device 100 via the interface 150 requesting the output of raw data. In step S5130, an example of a case where the estimated position of the control device 200 does not meet the conditions is when the reliability of the distance image data input from the distance measuring device 100 is low. In such a case, the processing circuit 230 requests the distance measuring device 100 to output raw data instead of distance data. As a result, in the next frame, raw data is input from the distance measuring device 100 to the control device 200 instead of distance data.

[0185] The subsequent steps S2180 to S2200 are the same as the corresponding operations in Figure 8. That is the case.

[0186] In this embodiment, the distance measuring device 100 outputs either distance image data or luminance image data for each exposure period. Alternatively, when the distance measuring device 100 receives a signal requesting luminance image data, it may output both distance image data and luminance image data for each exposure period. Also in this embodiment, the control device 200 always performs preprocessing and distance calculation when it receives luminance image data for each exposure period. Alternatively, the control device 200 may perform preprocessing and distance calculation only when necessary. The control device 200 may record and store the acquired luminance image data for each exposure period on the recording medium 270. Alternatively, the luminance image data for each exposure period may be stored by transmitting it to an external server or storage device via a network.

[0187] In this embodiment, the control device 200 determines whether or not to output a raw data request signal in step S5130 based on its own position. Alternatively, the control device 200 may determine whether or not to output a raw data request signal based on the state of the data acquired from the distance measuring device 100. For example, a raw data request signal may be output if the value of the high spatial frequency component of the acquired distance image data, or the proportion of high-frequency components in the whole, exceeds a predetermined threshold. A raw data request signal may also be output if the data acquired from multiple distance measuring devices 100 is inconsistent. Or, a raw data request signal may be output if the distance image data acquired from the distance measuring device 100 and the map data are inconsistent, such as when the point cloud is located inside a building shown in the map data. For example, a raw data request signal may be output if there are a predetermined number or more points in the point cloud corresponding to multiple pixels in the distance image data acquired from the distance measuring device 100 that are in positions inconsistent with the map data. Alternatively, if the control device 200 acquires image data separately, and there are more points between the acquired image data and the distance image data that cannot be matched than a predetermined number, it may output a raw data request signal.

[0188] Furthermore, the control device 200 may acquire measurement results from measuring devices other than the distance measuring device 100 and determine whether or not to output a raw data request signal based on the measurement results from the other measuring devices. The other measuring device may be a gyro sensor. For example, if the measurement result of the gyro sensor shows a steep change in the time direction, that is, if there is a large shake or impact, the control device 200 may output a raw data request signal to the distance measuring device 100. Alternatively, if the measurement result of the gyro sensor shows an angle above or below a predetermined threshold, that is, if the entire system is tilted, such as when driving on a steep slope, the control device 200 may output a raw data request signal to the distance measuring device 100. The other measuring device may also be a camera that takes brightness images or videos. For example, if a signal from the camera is acquired and there is a pixel in the frame of the acquired image or video that shows a brightness exceeding a predetermined value, the control device 200 may output a raw data request signal to the distance measuring device 100. Possible causes of such high-brightness pixels include the incidence of strong sunlight or the incidence of headlights from oncoming vehicles. Alternatively, a raw data request signal may be output to the distance measuring device 100 based on the analysis results of the motion vectors acquired from the video. For example, if the motion vector angle is unstable, it can be inferred that the system is being subjected to shock or vibration, and therefore a raw data request signal may be output.

[0189] If the system is a moving object, the control device 200 may determine whether or not to output a raw data request signal based on the operation plan of the moving object. If the operation plan is a specific operation such as acceleration / deceleration, lane change, or reversing, the control device 200 may output a raw data request signal to the distance measuring device 100. In addition, sensors from various operating parts of the moving object may detect brake lock and traction If a signal indicating an operational abnormality, such as a malfunction, is acquired, a raw data request signal may be output to the distance measuring device 100.

[0190] The processing circuit 230 of the control device 200 may decide whether or not to output a raw data request signal based on the communication status between the distance measuring device 100 and the control device 200. For example, if the memory (not shown) of the interface 210 stores more data than a predetermined amount, the processing circuit 230 may output a signal requesting distance image data. As an example, if the amount of data that the interface 210 receives per second exceeds or is expected to exceed a predetermined amount, the processing circuit 230 may output a signal requesting distance image data.

[0191] [effect] As described above, according to this embodiment, the processing circuit 130 of the distance measuring device 100 generates distance data for each pixel in addition to the brightness data for each pixel for each exposure period. Then, it switches between outputting the distance data and the brightness data for each exposure period in response to a request from the external control device 200. The processing circuit 130 decides for each frame whether to output a pair of brightness data for each exposure period and timing data indicating the timing of each exposure period, or the distance data. Furthermore, the processing circuit 230 of the control device 200 requests the distance measuring device 100 to output brightness data if the data sent from the distance measuring device 100 does not satisfy predetermined conditions.

[0192] This operation enables a system in which the distance measuring device 100 normally outputs distance data, and only outputs brightness data for each exposure period when higher accuracy distance measurement is required. The size of the brightness data for each exposure period is larger than the size of the distance data. According to the operation of this embodiment, the control device 200 can acquire the large brightness data for each pixel for each exposure period only when necessary. Therefore, the amount of communication between the control device 200 and the distance measuring device 100 can be reduced.

[0193] In this embodiment, under normal circumstances, the distance measuring device 100 outputs distance image data. In locations with complex traffic conditions, such as around intersections, the distance measuring device 100 can output luminance data for each exposure period according to instructions from the control device 200. The control device 200, upon receiving the luminance data for each exposure period from the distance measuring device 100, can generate more accurate distance information through preprocessing. Furthermore, by calculating the reflectance of each pixel, recognition performance can be improved, thereby enhancing the accuracy of environmental recognition. In addition, by recording the detailed luminance data for each exposure period directly or transferring it to other devices such as a server via a network, data indicating the state of a complex traffic environment can be retained. This allows for the recording of verification data that enables more detailed situation analysis, for example, in the event of an accident.

[0194] <Modification 1 of Embodiment 2> Next, a modification 1 of Embodiment 2 will be described. In this modification, the light source 110 emits a light beam with a smaller spread, rather than a flash of light. The light source 110 is controlled to change the direction of the light and project multiple times in order to project light over a wide area. The configuration of the light source 110 is the same as that in modification 1 of Embodiment 1.

[0195] Figure 24 is a flowchart showing the operation of the distance measuring device 100 in this modified example. In the operation shown in Figure 24, steps S3110 and S3120 are added to the operation shown in Figure 19, and step S1180 is replaced by step S3130. In all other respects, it is the same as the operation shown in Figure 19. Also, in the operation shown in Figure 24, step S1190 in the operation shown in Figure 13 is replaced by step S4150 instead of S4110. The operation of each step shown in Figure 24 is the same as the operation of the corresponding step shown in Figure 13 or Figure 19. Since they are similar, the explanation will be omitted.

[0196] As shown in this modified example, by using a light source 110 that can change the direction of light beam emission, and scanning the target scene with the light beam, it is possible to acquire data necessary for measuring the distance of distant objects compared to when using a light source 110 that emits flash light. Furthermore, by performing the operations from steps S4110 to S4140 shown in Figure 24, it is possible to switch between outputting distance data and outputting brightness data for each exposure period in response to a request from an external device.

[0197] Furthermore, the configuration of the light source 110 and the light receiving device 120 may be the same as that of Modification 2 of Embodiment 1. That is, a configuration may be adopted in which a light source 110 that emits a light beam and a light receiving device 120 equipped with one or a few light receiving elements are used, and the direction of the light source 110 and the light receiving device 120 is changed to acquire the light receiving data necessary for distance measurement. In such a configuration, distance data and brightness data for each exposure period may be switched and output.

[0198] <Modification 2 of Embodiment 2> Next, a modification 2 of Embodiment 2 will be described. In this modification, the content of the output data is switched based on the processing result of the processing circuit 130 of the distance measuring device 100, rather than based on a request from an external device such as the control device 200. The configuration of this modification is the same as that of Embodiment 2.

[0199] Figure 25 is a flowchart showing the operation of the processing circuit 130 in the distance measuring device 100 in this modified example. In the operation shown in Figure 25, steps S6110 and S6120 are executed instead of step S4110 in the operation shown in Figure 19. The steps other than steps S6110 and S6120 are the same as the corresponding steps shown in Figure 19. The differences from the operation shown in Figure 19 will be explained below.

[0200] (Step S6110) The processing circuit 130 processes the charge values ​​for each pixel during each exposure period, read from the charge storage unit 124, as two-dimensional array data according to the pixel arrangement. Random noise can be extracted as high-frequency components of the spatial frequency in the image. In this modified example, the processing circuit 130 first divides each of the luminance images for exposure periods A0, A1, and A2 into multiple regions. For example, each luminance image can be divided into eight regions by dividing it into two in the y-axis direction and four in the x-axis direction. Here, the luminance images for exposure periods A0, A1, and A2 are divided in the same way. The processing circuit 130 performs spatial frequency analysis on the two-dimensional data representing each divided region. For example, a two-dimensional Fourier transform can be used as a method for frequency analysis. The processing circuit 130 extracts the real part of each frequency component obtained as a result of the Fourier transform. The processing circuit 130 calculates the integral of the absolute value of the extracted frequency components that are above a predetermined frequency and takes this as the noise amount. The processing circuit 130 calculates the above noise amount for each divided region of the three luminance images corresponding to exposure periods A0, A1, and A2, respectively.

[0201] (Step S6120) The processing circuit 130 evaluates the noise level based on the noise level in each segmented region of each luminance image calculated in step S6110. If the noise level exceeds a threshold in step S6120, the process proceeds to step S4140. If the noise level does not exceed a threshold in step S6120, the process proceeds to step S4120. As an example of how to evaluate the noise level, the processing circuit 130 may determine that there is a lot of noise if the sum of the noise levels in the entire image is greater than a predetermined value in any of the luminance images for exposure periods A0, A1, and A2. As another example of how to determine the noise level, the processing circuit 130 may determine that there is a lot of noise if the sum of the noise levels in the entire image is greater than a predetermined value. A high level of noise may be determined if the noise level in some of the divided regions is greater than a predetermined value. For example, if the noise level in one of the eight divided regions is greater than a predetermined value, a high level of noise may be determined. A high level of noise may also be determined if the noise level is greater than a predetermined value in two or more regions. In this way, by dividing each luminance image into multiple regions and calculating and determining the noise level for each region, even if there is high noise in some regions of the image, it can be detected without being overlooked.

[0202] In the above method, the processing circuit 130 may determine that "the amount of noise has exceeded the threshold" if it determines that there is a lot of noise in any of the brightness images for exposure periods A0, A1, and A2. The method for determining the amount of noise is not limited to the method described above, but may be other methods as well.

[0203] In step S6120, if it is determined that the noise level exceeds the threshold, the processing circuit 130 outputs luminance image data, i.e., raw data, via the interface 150 (S4140). On the other hand, if it is determined that the noise level does not exceed the threshold, the processing circuit 130 generates and outputs distance image data by calculating the distance for each pixel using the method described above (steps S4120 and S4130).

[0204] The processing circuit 130 can switch between outputting distance image data and outputting luminance image data for each exposure period, depending on the state of the acquired luminance data, by repeating the operation from step S1120 to step S4150. In this modified example, the amount of noise that can be evaluated based on the magnitude of high-frequency components is used as the state of the luminance data, but it may also be evaluated based on other indicators related to the reliability of the luminance data. For example, a similar determination may be made based on the reflectance of each pixel calculated from the luminance data of each pixel and the maximum value of the luminance value when the reflectance for each distance is 100%, which has been recorded in advance. The processing circuit 130 may output luminance image data for each exposure period instead of distance image data if the reflectance index value calculated based on the reflectance of one or more pixels in the three luminance images corresponding to exposure periods A0, A1, and A2, respectively, is lower than a predetermined value. Alternatively, if a predetermined proportion of pixels have luminance values ​​that exceed a predetermined threshold, the processing circuit 130 may output luminance image data for each exposure period instead of distance image data.

[0205] In this modified example, the processing circuit 130 determines in steps S6110 and S6120 whether or not to output luminance image data, depending on the state of the acquired luminance data. Alternatively, the processing circuit 130 may output luminance image data for each exposure period if the distance measuring device 100 malfunctions for any reason. The malfunction of the distance measuring device 100 may be detected by other measuring devices such as a gyro sensor.

[0206] To enable the output of luminance image data when the distance measuring device 100 malfunctions, the recording medium 170 stores the most recent luminance image data for one or more frames along with time data. When the distance measuring device 100 malfunctions, the processing circuit 130 reads the luminance image data for each exposure period stored in the recording medium 170, working backward from the time of the malfunction, and outputs it to the control device 200. The output destination may be not only the control device 200, but also an external system such as a traffic information center via communication.

[0207] In this modified example, a light source 110 that emits flash light is used, but as in Modification Example 1 described above, a beam scanner that emits a narrowly spreading light beam may be used as the light source 110. Alternatively, as in Modification Example 2 of Embodiment 1, a light source that emits a narrowly spreading light beam and a light receiving device equipped with one or a few light receiving elements may be used.

[0208] The data formats in the examples above are illustrative, and the distance measuring device 100 can also measure data in other formats. The device may output information in any format. The distance measuring device 100 may also reduce the amount of data by generating luminance data or distance image data in any of the above formats and then compressing the data using a predetermined method.

[0209] When multiple distance measuring devices 100 transmit data to a single control device 200, an identifier for identifying the distance measuring device 100 may be added to the data included in the data format in each of the above examples.

[0210] When multiple distance measuring devices 100 output raw data to a single control device 200, the control device 200 may process the raw data from the multiple distance measuring devices 100 together. Such processing may include, for example, addition, averaging, or filtering. By processing the raw data output from multiple distance measuring devices 100 together, the accuracy of distance calculation can be improved.

[0211] <Modification 3 of Embodiment 2> Next, a modification 3 of Embodiment 2 will be described. In this modification, the processing circuit 130 of the distance measuring device 100 decides whether to output luminance data or distance data for each region of the image, rather than on a frame-by-frame basis. In this modification, the processing circuit 130 divides the pixel group of the image sensor 121 into multiple regions and switches between outputting distance data and luminance data for each exposure period for each region. Therefore, the output data for each frame in this modification includes both raw data and distance data. The output data includes data that identifies each region and data that indicates whether each region is output as raw data or distance data. After receiving data from the distance measuring device 100, the processing circuit 230 of the control device 200 similarly divides the image indicated by the data into multiple regions, determines whether each region is raw data or distance data, and performs different processing according to the determination result.

[0212] The system configuration in this modified example is the same as that shown in Figure 18. However, the data recorded on the recording medium 270 and the operation of the processing circuits 130 and 230 differ from the example in Figure 18.

[0213] Figure 26 is a schematic diagram showing an example of data recorded on the recording medium 270. In this example, when the control device 200 receives image data from the distance measuring device 100, data as shown in Figure 26 is recorded on the recording medium 270. In the example shown in Figure 26, the date, area ID, and area range are recorded, along with the area ID, distance, brightness during exposure period A0, brightness during exposure period A1, and brightness during exposure period A3, which are recorded for each pixel. For pixels where distance data is recorded, the brightness values ​​for exposure periods A0, A1, and A2 are not recorded and remain blank. Conversely, for pixels where brightness values ​​for exposure periods A0, A1, and A2 are recorded, the distance is not recorded and remains blank. After acquiring the data transmitted from the distance measuring device 100, the processing circuit 230 sequentially calculates and records the distance from the brightness value data for exposure periods A0, A1, and A2 for pixels where distance values ​​are not recorded. As a result, as processing progresses, blank distance data is sequentially replaced with calculated distance data.

[0214] [Operation of the rangefinder] Figure 27 is a flowchart showing the operation of the distance measuring device 100 in this modified example. The operation shown in Figure 27 is the same as the operation shown in Figure 19, except that steps S4110 to S4150 are replaced with steps S7110 to S7170. The differences from the operation shown in Figure 19 will be explained below.

[0215] (Step S7110) Once the acquisition of luminance data for each exposure period of one frame is complete through the operations in steps S1120 to S1180, the processing circuit 130 divides each image indicated by the data into multiple regions. At this time, each image is divided into multiple regions according to the region ID and region range information described in the signal transmitted from the control device 200.

[0216] (Step S7120) The processing circuit 130 determines whether the generation of output data has ended for all of the regions divided in step S7110. If the generation of output data has ended for all regions, the process proceeds to step S7140. If there are still regions for which the generation of output data has not ended, the process proceeds to step S7130.

[0217] (Step S7130) The processing circuit 130 selects one of the regions among those divided in step S7110 for which output data has not yet been generated.

[0218] (Step S7140) The processing circuit 130 determines, based on the signal transmitted from the control device 200, whether the output of load data is requested as the output data for the region selected in step S7130. If the output of load data is requested for the region, the process returns to step S7120. If the output of load data is not requested for the region, the process proceeds to step S7150.

[0219] (Step S7150) The processing circuit 130 calculates the distance for each pixel in the region selected in step S7130 based on the luminance value for each exposure period by the method described above.

[0220] (Step S7160) The processing circuit 130 records the distance for each pixel calculated in step S7150 in the recording medium 170.

[0221] FIG. 28 is a diagram showing an example of the data recorded in the recording medium 170. A pair of a time ID and a detailed time, and a pair of a region ID and a region range are recorded. Also, for each pixel, the ID of the region containing the pixel, the value of the charge in each exposure period read in step S1180, and the time ID specifying each exposure period are recorded. For pixels in a region for which the output of load data is not requested, the distance calculated in step S7150 is also recorded. For pixels in a region for which the output of load data is requested, the distance is not recorded.

[0222] (Step S7170) The processing circuit 130 converts the distance recorded for each pixel into a pixel value to generate distance data. For pixels without distance data, it converts the charge value for each exposure period into a brightness value to generate brightness data. The processing circuit 130 outputs the generated image data along with detailed time data via the interface 150.

[0223] [Example of data format] Figures 29A and 29B show examples of the output data format. In this example, the position, orientation, field of view, pixel arrangement, and timing data for each exposure period of the rangefinder are output as fixed values ​​common to multiple frames, similar to the examples in Figures 7A and 7B.

[0224] In the examples in Figures 29A and 29B, the following data is output sequentially as values ​​that vary from frame to frame. First, the date and detailed time when the data was acquired, and the image sensor 12 The number of regions in which the pixel array is divided is output. Next, the starting pixel coordinates and ending pixel coordinates indicating the range of each region are output for each region. In this modified example, the shape of each region is rectangular, but other shapes such as ellipses may also be output. Following this, the number of regions for which distance data is output, the ID of each region for which distance data is output, the number of regions for which raw data is output, and the ID of each region for which raw data is output are output. Following this information, the distance of each pixel for which raw data output is not requested is output. Furthermore, for each exposure period, the brightness value of each pixel for which raw data output is requested is output in order.

[0225] [Control device operation] Figure 30 is a flowchart showing an example of the processing performed by the processing circuit 230 of the control device 200 in this modified example. Of the steps shown in Figure 30, steps S2120, S2130, and S2170-S2200 are the same as the corresponding steps shown in Figure 8. The operation of each step will be described below.

[0226] (Step S2120) The processing circuit 230 determines whether or not an operation termination signal has been input from an external device (not shown). If an operation termination signal has been input, the operation is terminated. If no operation termination signal has been input, the process proceeds to step S2130.

[0227] (Step S2130) The processing circuit 230 determines whether or not there is data input from the distance measuring device 100. If there is data input from the distance measuring device 100, the data is recorded on the recording medium 270 and the process proceeds to step S9110. If there is no data input from the distance measuring device 100, the process returns to step S2120.

[0228] (Step S9110) The processing circuit 230 obtains data from the input data for each frame recorded on the recording medium 270, indicating the number of regions to divide the image represented by the data into multiple regions, and the pixel range of each region. Furthermore, the processing circuit 230 determines whether processing has been completed for all regions. If processing has been completed for all regions, the process proceeds to steps S9170 and S2170. If there are unprocessed regions among the divided multiple regions, the process proceeds to step S9120. In this modified example, the processing from steps S9170 to S9190 and the processing from steps S2170 to S2200 are performed in parallel, but these may also be performed in series.

[0229] (Step S9120) The processing circuit 230 selects one of the divided regions that has not yet been processed.

[0230] (Step S9130) The processing circuit 230 refers to the input data for each frame recorded on the recording medium 270 and determines whether the region ID of the region selected in step S9120 is included in the distance region ID. If the region ID of the region is included in the distance region ID, the process returns to step S9110. If the region ID of the region is not included in the distance region ID, i.e., if the region ID of the region is included in the Raw region ID, the process proceeds to step S9140.

[0231] (Step S9140) The processing circuit 230 acquires the raw data of the relevant region from the input data from the distance measuring device 100 recorded on the recording medium 270, i.e., the luminance data for exposure periods A0, A1, and A2 shown in Figures 29A and 29B. The processing circuit 230 then processes the acquired raw data. Preprocessing is performed on the data. As an example of preprocessing, noise reduction is performed. Noise reduction is performed using adaptive filters on the luminance image of the pixel range extracted as the region during exposure period A0, the luminance image of exposure period A1, and the luminance image of exposure period A2. Other noise reduction processes besides adaptive filters may also be performed as preprocessing. In addition, signal processing other than noise reduction, such as contrast enhancement or edge extraction, may also be performed.

[0232] (Step S9150) The processing circuit 230 calculates the distance for each pixel using the image data for each exposure period, which has undergone noise reduction processing in step S9140, for the pixel range included in the region. The pixel values ​​of identical pixels can be extracted from the image data for each exposure period within the region, and the distance can be determined based on the calculation formula described above.

[0233] (Step S9160) The processing circuit 230 stores the distance data for each pixel within the region, calculated in step S9150, in the recording medium 270. After executing step S9160, the process returns to step S9110.

[0234] (Step S9170) For the entire area, when distance data is generated, the processing circuit 230 combines the distance of the pixel having a distance value in the data input from the distance measuring device 100 with the distance of the pixel whose distance is calculated in step S9150 to generate distance image data. Then, the pixels of this distance image data are clustered according to the distance. As a result, each pixel in the distance image data is classified into one of a plurality of clusters corresponding to a plurality of distance ranges respectively.

[0235] (Step S9180) For each of the clusters generated in step S9170, excluding the clusters of distance 0 and infinite distance, the processing circuit 230 extracts the area around the cluster where the ratio N1 / N2 of the number of pixels N1 within the distance range characterizing the cluster to the number of pixels N2 outside the distance range characterizing the cluster is not less than a first threshold value and less than a second threshold value larger than the first threshold value. The processing circuit 230 sets the rectangular area closest to the extracted area as the load data request area.

[0236] (Step S9190) The processing circuit 230 divides the area other than the load data request area set in step S9180 into one or more rectangular areas, and sets each rectangular area as a non-load data request area, that is, a distance data request area. For each of the load data request area and the distance data request area, the processing circuit 230 transmits an instruction signal including data indicating the range of the area and a data format specifying the area, that is, a data code indicating whether it is load data or distance data, to the distance measuring device 100.

[0237] FIG. 31 is a diagram showing an example of the format of the instruction signal. In the example of FIG. 31, the instruction signal includes data indicating the number of areas, a data code (that is, a binary value representing load data / distance data), the range of each area, and a data code described in the order of area IDs.

[0238] The processing circuit 230 generates an output signal to the distance measuring device 100 through the operation of steps S9170 to S9190. In parallel, the processing circuit 230 generates a control signal for the autonomous vehicle based on the output of the distance measuring device 100 through the operation of steps S2170 to S2200. The operation of steps S2170 to S2200 is the same as the operation of the corresponding steps shown in Figure 8. After steps S9190 and S2200, step The process returns to S2120, and the same actions are repeated.

[0239] [effect] As described above, in this modified example, the processing circuit 230 of the control device 200 divides the pixel group in the image sensor 121 of the distance measuring device 100 into multiple regions based on the distance image data generated based on the output of the distance measuring device 100. Then, it requests the distance measuring device 100 to output either raw data or distance data for each region. In response to the request from the control device 200, the processing circuit 130 of the distance measuring device 100 decides for each region whether to output raw data or distance data. This allows the processing circuit 230 of the control device 200 to perform more detailed signal processing by requesting the output of raw data for regions where the accuracy of the distance data output from the distance measuring device 100 is low. As a result, the processing circuit 230 can generate highly accurate distance data that would be difficult for the processing circuit 130 of the distance measuring device 100, which has relatively low performance, to generate in a short time.

[0240] In this modified example, the control device 200 requests the rangefinder 100 to output raw data or distance data for each region. Alternatively, for example, the rangefinder 100 may be instructed to output distance data for all pixels, and then, for specific regions, it may be requested to output raw data in addition to distance data. When requesting additional raw data output for specific regions, the instruction signal output from the control device 200 may include, for example, data specifying the number of regions for which additional raw data should be output and the respective ranges of those regions.

[0241] Figure 32 is a flowchart showing an example of the operation of the distance measuring device 100 when an instruction signal is received requesting the output of additional raw data for a specific area. In the operation shown in Figure 32, steps S1110 to S1180, S7150 and S7160 are the same as the corresponding steps shown in Figure 27. In the example shown in Figure 32, steps S7210 and S7220 are performed instead of steps S7110, S7120, S7130 and S7240 in the example in Figure 27. Also, step S7230 is performed instead of step S7170. These steps will be explained below.

[0242] (Step S7210) After the operation of step S1180, the processing circuit 130 records on the recording medium 170 the area in which raw data is output in addition to distance data, as described in the instruction signal transmitted from the control device 200.

[0243] (Step S7220) The processing circuit 130 determines whether distance calculation has been completed for all pixels of the image sensor 121. If distance calculation has been completed for all pixels, the process proceeds to step S7230. If there are pixels for which distance calculation has not been completed, the process proceeds to steps S7150 and S7160, and then returns to step S7220.

[0244] (Step S7230) The processing circuit 130 outputs distance data for each pixel recorded on the recording medium 170, and raw data for each pixel in the area specified by the instruction signal transmitted from the control device 200.

[0245] Figures 33A and 33B show an example of the output format in this modified example. In this example, following a fixed value, for each frame, the date and time, time, distance calculated for all pixels, number of areas where raw data is output, range of each area where raw data is output, and brightness indicating the charge accumulated for each pixel in exposure periods A0, A1, and A2 are output. In this way, the distance measuring device 100 provides distance information for all pixels in addition to... Furthermore, when outputting additional raw data for pixels within a specific region, data in the format shown in Figures 33A and 33B may be output.

[0246] Figure 34 shows an example of the operation of the processing circuit 230 when the control device 200 receives data output in the format shown in Figures 33A and 33B. The operation shown in Figure 34 is the same as the operation shown in Figure 30, except that steps S9110 to S9160 are replaced with steps S9310 to S9340. The following mainly describes the operation that differs from the example in Figure 30.

[0247] (Step S9310) The processing circuit 230 identifies the region containing raw data output and its pixel range from the input data for each frame recorded on the recording medium 270, and determines whether processing has been completed for all of the regions containing raw data output. If processing has been completed for all of the regions containing raw data output, the process proceeds to steps S9170 and S2170. If there are unprocessed regions among the regions containing raw data output, the process proceeds to step S9320.

[0248] (Step S9320) The processing circuit 230 refers to the input data for each frame recorded on the recording medium 270 and selects one of the areas with raw data output that has not yet been processed.

[0249] (Step S9140) The processing circuit 230 acquires the brightness values ​​for exposure periods A0, A1, and A2 for each pixel within the range of the region selected in step S9320. The processing circuit 230 then performs preprocessing on the acquired brightness values ​​for each exposure period. Preprocessing is, for example, the noise reduction process described above.

[0250] (Step S9150) The processing circuit 230 calculates the distance for each pixel in the region using the image data for each exposure period after the noise reduction process in step S9140.

[0251] (Step S9330) The processing circuit 230 compares the variance of the pixel-by-pixel distance data output from the distance measuring device 100 with the variance of the distance data calculated in step S9150 after noise reduction processing in step S9140 for the pixel range included in the region. If the variance of the pixel-by-pixel distance data calculated in step S9150 is smaller than the variance of the distance data output from the distance measuring device, the process proceeds to step S9340. If the variance of the pixel-by-pixel distance data calculated in step S9150 is greater than or equal to the variance of the distance data output from the distance measuring device, the process returns to step S9310.

[0252] (Step S9340) The processing circuit 230 replaces the distance values ​​for each pixel in the region recorded on the recording medium 170 with the distance values ​​calculated in step S9150. This corrects the distance data for the region to distance data with reduced noise.

[0253] In this modified example, the control device 200 clusters the distance image according to distance range and divides the image into multiple regions based on the distribution of distance data of the surrounding pixels of the cluster, but the multiple regions may be determined by other methods. The range of each region may not change from frame to frame but may be fixed. Also, based on the distribution of distance data or the magnitude of noise, The area may be set accordingly. Alternatively, the area setting may be changed according to the speed of the moving object on which the distance measuring device 100 and control device 200 are mounted. Furthermore, the area may be set by methods other than those mentioned above.

[0254] As shown in this modified example, when data of different formats are mixed within a single frame, data compression becomes difficult. Therefore, when compressing output data in which distance data and raw data are mixed within a single frame, the distance data and raw data may be separated and compressed. The control device 200, which receives data transmitted from the distance measuring device 100, may process the acquired dataset containing distance data and raw data, and compress all pixels by converting them to the distance data format.

[0255] In this modified example, the light source 110 of the distance measuring device 100 emits flash light, but as in Modification 1 of Embodiment 1, a beam scanner that emits a light beam with a small divergence angle may be used as the light source 110. When a beam scanner is used, the movement of the light emission direction from the light source 110 can be performed in the same manner as in Modification 1 of Embodiment 1. The timing of light emission and reception, and the repetition of exposure associated with the scanning operation can be the same as in Modification 1 of Embodiment 1. Regarding the output data, in order to address the shift in measurement time due to performing distance measurement over a wide area by scanning with a light beam with a small divergence angle, the output data may include detailed time data for each direction of the light beam.

[0256] In the embodiments described above, the operation of the indirect ToF method, in which light-receiving data is acquired for each of the three exposure periods, has been mainly explained. However, this disclosure is not limited to such operation. For example, the number of exposure periods is not limited to three, but may be two or four or more. [Industrial applicability]

[0257] The technology disclosed herein is widely applicable to devices or systems that perform ranging. For example, the technology disclosed herein can be used as a component of a LiDAR (Light Detection and Ranging) system. [Explanation of Symbols]

[0258] 100 Rangefinder 110 Light source 120 Photodetector 121 Image Sensor 122 Photodetector 123 Switch 124 Charge storage section 130 Processing Circuits 150 interfaces 160 clock 170 Recording media 200 Vehicle control system 210 Interfaces 230 Processing Circuits 270 recording media 240 Operating part

Claims

1. Image sensor and Processing circuit and Equipped with, The aforementioned processing circuit is The image sensor receives reflected light from the scene during each of the multiple exposure periods, thereby acquiring pixel value data corresponding to each of the multiple exposure periods. Distance data is generated based on the aforementioned pixel value data. The distance data and the pixel value data corresponding to each of the multiple exposure periods are switched and output. Sensing device.

2. The processing circuit switches between outputting the distance data and the pixel value data in response to a request from an external device. The sensing device according to claim 1.

3. The processing circuit switches between outputting the distance data and the pixel value data according to the state of the pixel value data. The sensing device according to claim 1 or 2.

4. The aforementioned processing circuit is The noise amount of the pixel value data for at least one of the plurality of exposure periods is calculated. When the noise level exceeds the threshold, the pixel value data is output. When the noise amount does not exceed the threshold, the distance data is output. A sensing device according to any one of claims 1 to 3.

5. The aforementioned processing circuit is The reflectance is calculated from the pixel value data for at least one of the plurality of exposure periods. When the reflectance exceeds the threshold, the distance data is output. When the reflectance does not exceed the threshold, the pixel value data is output. A sensing device according to any one of claims 1 to 3.

6. The sensing device according to claim 1, wherein when the processing circuit outputs the pixel value data or the distance data, it outputs the pixel value data or the distance data with an identifier indicating whether it contains the pixel value data or the distance data.

7. The sensing device according to any one of claims 1 to 6, wherein the processing circuit switches and outputs the distance data and the pixel value data for each of the multiple regions included in the scene.

Images