Measuring and ranging devices

Abstract

Provided is a measuring device (1) according to the present disclosure. The measuring device (1) comprises a plurality of photodetectors and a recognition unit (111). The plurality of photodetectors are disposed on a first substrate (P1) and output signals upon receiving light emitted from a light source and reflected off an object being measured. The recognition unit (111) is disposed on a second substrate (P3), which is different from the first substrate (P1), and uses a machine learning model to recognize information about the distance to the object being measured on the basis of the signals outputted from the plurality of photodetectors.

Description

[Technical field]

[0001] The present disclosure relates to a measurement device and a ranging device. [Background technology]

[0002] A direct ToF (direct Time of Flight) method is known as one of the distance measurement methods that uses light to measure the distance to an object. In the direct ToF method, the distance to the object is calculated based on the time from the emission timing indicating the emission of light from a light source to the light reception timing when the reflected light reflected by the object is received by a light receiving element.

[0003] More specifically, the time from the emission timing to the light reception timing at which the light is received by the light receiving element is measured, and time information indicating the measured time is stored in memory. This measurement is performed multiple times, and a histogram is created based on the multiple pieces of time information from the multiple measurements stored in memory. The distance to the object is calculated based on this histogram. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-176750 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above conventional technology requires various processes to generate a histogram, to detect time information by thresholding the histogram, and to obtain the distance to the object, which results in a problem of long processing time. In addition, there is a problem that it is time-consuming to adjust parameters required for various processes. Furthermore, there is a problem that it is difficult to improve the accuracy of distance calculation because it depends on the memory capacity of the memory that stores the time information and the number of bins of the histogram.

[0006] Therefore, the present disclosure proposes a measurement device and a distance measuring device that can reduce processing time and the effort required for parameter adjustment and improve accuracy. [Means for solving the problem]

[0007] According to the present disclosure, a measurement device is provided. The measurement device includes a plurality of light receiving elements and a recognition unit. The plurality of light receiving elements are disposed on a first substrate, and output a signal when light irradiated from a light source and reflected by an object to be measured is received. The recognition unit is disposed on a second substrate different from the first substrate, and recognizes information regarding the distance to the object to be measured using a machine learning model based on the output signals of the plurality of light receiving elements. Effect of the Invention

[0008] According to the present disclosure, it is possible to reduce the processing time and the effort required for parameter adjustment, and to improve accuracy. Note that the effects described herein are not necessarily limited, and may be any of the effects described in the present disclosure. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of distance measurement using a direct ToF method. [Diagram 2] FIG. 13 is a diagram showing an example of a histogram based on the time when the light receiving unit receives light. [Diagram 3] FIG. 1 is a diagram for explaining an overview of a distance measuring method according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram showing an example of a configuration of a distance measuring system according to an embodiment of the present disclosure. [Diagram 5] 1 is a block diagram showing an example of a configuration of a measurement device according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a layered structure of a measurement device according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a diagram for explaining an overview of a neural network. [Figure 8] FIG. 1 is a diagram for explaining an example of a machine learning model. [Figure 9] FIG. 1 is a diagram for explaining an example of a machine learning model. [Figure 10] 11 is a diagram for explaining recognition of time information by a recognition unit; FIG. [Figure 11] FIG. 2 is a block diagram showing a configuration example of a control device according to an embodiment of the present disclosure. [Figure 12] 11 is a diagram for explaining a distance measurement position determined by a distance measurement determination unit; FIG. [Figure 13] 11 is a diagram for explaining a distance measurement position determined by a distance measurement determination unit; FIG. [Figure 14] 11 is a flowchart illustrating an example of a procedure of a model selection process according to an embodiment of the present disclosure. [Figure 15] 11 is a flowchart illustrating an example of a procedure of a measurement process according to an embodiment of the present disclosure. [Figure 16] FIG. 13 is a diagram illustrating an example of a distribution of time information of pixels. [Figure 17] FIG. 13 is a diagram showing another example of the distribution of time information of pixels. [Figure 18] 1A to 1C are diagrams showing examples of use of a measurement device to which the above-described embodiments and modifications can be applied. [Figure 19] 1 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile object control system to which the technology according to the present disclosure can be applied. [Figure 20] FIG. 4 is a diagram showing an example of an installation position of an imaging unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, each embodiment of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] (Distance measurement method using histogram) Prior to describing the embodiments of the present disclosure, in order to facilitate understanding, a technique for measuring distance using a histogram will be described as one of the distance measurement methods. In this case, a direct ToF (direct time of flight) method is applied as the distance measurement technique. The direct ToF method is a technique in which light emitted from a light source is reflected by an object to be measured and received by a light receiving element, and distance measurement is performed based on the time difference between the light emission timing and the light reception timing.

[0012] An example of a distance measurement method using the direct ToF method will be described briefly with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram showing an example of distance measurement using the direct ToF method. The distance measurement device 300a includes a light source unit 301a and a light receiving unit 302a. The light source unit 301a is, for example, a laser diode, and is driven to emit a laser light in a pulsed form. The light emitted from the light source unit 301a is reflected by the object to be measured 303a and is received by the light receiving unit 302a as reflected light. The light receiving unit 302a includes a light receiving element that converts light into an electrical signal by photoelectric conversion, and outputs a signal according to the received light.

[0013] Here, the time when the light source unit 301a emits light (light emission timing) is time t em The time (light receiving timing) at which the light receiving unit 302a receives the light emitted from the light source unit 301a and reflected by the object 303a is defined as time t re Let the constant c be the speed of light (2.9979 × 10 8 [m / sec], the distance D between the distance measuring device 300a and the object to be measured 303a is calculated by the following equation (1). D = (c / 2) × (t em -tre ) …(1)

[0014] The distance measuring device 300a repeats the above process multiple times. The light receiving unit 302a may include multiple light receiving elements, and the distance D may be calculated based on the light receiving timing at which the reflected light is received by each light receiving element. em The time t from the timing when the light is received by the light receiving unit 302a m (Light receiving time t m The data (called "data") are classified into classes (bins) to generate a histogram.

[0015] In addition, the light receiving unit 302a receives light for a period of time t m The light received by the light receiving unit 302a is not limited to light emitted by the light source unit 301a and reflected by the object to be measured 303a. For example, ambient light around the distance measuring device 300a (light receiving unit 302a) is also received by the light receiving unit 302a.

[0016] 2 is a diagram showing an example of a histogram based on the time when the light receiving unit 302a receives light. In FIG. 2, the horizontal axis indicates bins, and the vertical axis indicates the frequency of each bin. A bin is a frequency of light receiving time t m are classified for each unit time d. Specifically, bin #0 is 0≦t m <d、ビン#1がd≦t m < 2 × d, bin #2 is 2 × d ≦ t m < 3 × d, …, bin #(N-2) is (N-2) × d ≦ t m <(N-1)×d. The exposure time of the light receiving unit 302a is time t ep In this case, t ep = N × d.

[0017] The distance measuring device 300a has a light reception time t mThe number of times the number of times is acquired is counted based on the bin to obtain the frequency 310a for each bin, and a histogram is generated. Here, the light receiving unit 302a receives light other than the reflected light that is the light emitted from the light source unit 301a. An example of such light other than the target reflected light is the above-mentioned ambient light. The part indicated by the range 311a in the histogram includes the ambient light component due to the ambient light. The ambient light is light that is randomly incident on the light receiving unit 302a, and becomes noise in the target reflected light.

[0018] On the other hand, the reflected light of interest is light received according to a specific distance, and appears as an active light component 312a in the histogram. The bin corresponding to the frequency of the peak in this active light component 312a is the bin corresponding to the distance D of the object 303a. The distance measuring device 300a calculates the representative time of the bin (for example, the central time of the bin) as the above-mentioned time t re By acquiring the light receiving results as above, the distance D to the object 303a to be measured can be calculated according to the above-mentioned formula (1). In this way, by using a plurality of light receiving results, it becomes possible to perform appropriate distance measurement against random noise.

[0019] However, in the distance measurement method using the histogram described above, the light receiving timing, t re It is necessary to obtain multiple light reception results before detecting the light reception timing, and the detection process takes time. m This requires memory to store the time t re In order to properly detect, it is necessary to set an appropriate threshold for each generated histogram, and since filtering and threshold setting processes are required, the processing load increases. In addition, depending on the required distance accuracy, it is necessary to select a filter coefficient that can reliably remove noise and a filter coefficient that can reliably remove noise during the time t re Since thresholds are required to properly detect the above, the development effort required to set these increases.

[0020] Thus, the distance measurement method using the histogram is desired to be improved from the viewpoint of reducing the processing time, the processing load, or the memory. Therefore, in the technology disclosed herein, a histogram is not generated, but the light receiving timing (time t re The distance to the object is recognized from the object. The outline of this technology will be explained below with reference to FIG.

[0021] (Embodiment) [Outline of distance measurement method] Fig. 3 is a diagram for explaining an overview of a distance measuring method according to an embodiment of the present disclosure. The distance measuring method shown in Fig. 3 is executed by a measuring device 1 (not shown in Fig. 3). The measuring device 1 also has a light receiving unit 302 having a plurality of pixels (light receiving elements).

[0022] As shown in Fig. 3, the light receiving unit 302 of the measurement device 1 outputs each light receiving timing at which the reflected light from the object is received for each light receiving element (step S1). Based on the distribution 310 of the light receiving timing, the measurement device 1 calculates a time t corresponding to the distance to the object using a machine learning model. re Specifically, the measurement device 1 recognizes a time t corresponding to the distance to the object to be measured using a distribution of light receiving timings by a plurality of light receiving elements as input data. re The machine learning model M is trained in advance using the correct answer data. The measurement device 1 inputs the light receiving timing output by the light receiving unit 302 in step S1 as input data to the machine learning model M, and calculates the output result of the machine learning model M based on the time t re Obtain as.

[0023] The measuring device 1 measures the acquired time t re Then, data including the distance to the object to be measured is generated (step S3).

[0024] In this way, the measuring device 1 calculates the time t corresponding to the distance to the object to be measured using a machine learning model that uses the light receiving timing as an input. reBy recognizing the distance, the measurement device 1 does not need to generate a histogram, and the time required to measure the distance can be shortened. Furthermore, the processing required for generating a histogram is no longer necessary, and the measurement device 1 can reduce the processing load for measuring the distance. Furthermore, the memory for generating a histogram is no longer necessary, and the circuit scale of the measurement device 1 can be reduced.

[0025] [Configuration of distance measurement system] 4 is a block diagram showing an example of a configuration of a distance measuring system according to an embodiment of the present disclosure. The distance measuring system 6 shown in FIG. 4 includes a measurement device 1, a light source 2, a storage device 7, a control device 4, an optical system 5, and an imaging device 8.

[0026] The light source 2 is, for example, a laser diode, and is driven to emit a pulsed laser light. The light source 2 may be a VCSEL (Vertical Cavity Surface Emitting LASER) that emits a laser light as a surface light source. The light source 2 may be an array in which laser diodes are arranged in a line, and a configuration in which the laser light emitted from the laser diode array is scanned in a direction perpendicular to the line may be applied. Furthermore, a laser diode may be used as a single light source, and a configuration in which the laser light emitted from the laser diode is scanned in the horizontal and vertical directions may be applied.

[0027] The measurement device 1 includes a plurality of light receiving elements. The plurality of light receiving elements are arranged, for example, in a two-dimensional lattice pattern to form a light receiving surface. The measurement device 1 calculates a time t corresponding to the distance to the object to be measured by using machine learning based on the output signals of the plurality of light receiving elements. re and generates distance data. The calculated distance data is stored in, for example, a storage device 7. The optical system 5 guides light incident from the outside to a light receiving surface included in the measurement device 1. Note that hereinafter, an apparatus including the light source 2 and the measurement device 1 is also referred to as a distance measuring apparatus.

[0028] The imaging device 8 is, for example, an RGB camera that captures an RGB image of a subject space in which an object to be measured exists.

[0029] The control device 4 controls the overall operation of the distance measuring system 6. For example, the control device 4 supplies the measurement device 1 with a light emission trigger signal that is a trigger for causing the light source 2 to emit light. The measurement device 1 causes the light source 2 to emit light at a timing based on the light emission trigger signal, and also outputs a time t em In addition, the control device 4 sets a pattern for distance measurement in the measurement device 1 in response to, for example, an external instruction.

[0030] Furthermore, the control device 4 controls switching of the machine learning model M used for machine learning by the measuring device 1, based on the RGB image captured by the imaging device 8. The measuring device 1 selects the machine learning model M according to, for example, the type of the object to be measured, according to the control of the control device 4.

[0031] [Measurement device configuration] 5 is a block diagram showing an example of a configuration of the measurement device 1 according to the embodiment of the present disclosure. As shown in FIG. 5, the measurement device 1 includes a pixel array unit 100, a distance measurement processing unit 101, a pixel control unit 102, an overall control unit 103, a clock generation unit 104, a light emission control unit 105, an interface (I / F) unit 106, and a model storage unit 107.

[0032] The overall control unit 103 controls the overall operation of the measuring device 1 according to, for example, a pre-installed program. The overall control unit 103 can also execute control according to an external control signal supplied from the outside. The clock generation unit 104 generates one or more clock signals used in the measuring device 1 based on a reference clock signal supplied from the outside. The light emission control unit 105 generates a light emission control signal indicating the light emission timing according to a light emission trigger signal supplied from the outside. The light emission control signal is supplied to the light source 2 and also to the distance measurement processing unit 101.

[0033] The pixel array unit 100 is a light receiving unit including a plurality of pixels 10, 10, ... each including a light receiving element, which are arranged in a two-dimensional lattice. The operation of each pixel 10 is controlled by a pixel control unit 102 in accordance with instructions from an overall control unit 103. For example, the pixel control unit 102 controls each pixel 10 so that the output signal of each pixel 10 is read out at once. Furthermore, the pixel 10 has, for example, a SPAD (Single Photon Avalanche Diode) element as a light receiving element.

[0034] The distance measurement processing unit 101 measures the distance D to the object based on the output signal read out from each pixel 10. The distance measurement processing unit 101 includes a conversion unit 110, a recognition unit 111, a data generation unit 112, and a model switching unit 113.

[0035] The conversion unit 110 converts the output signal supplied from the pixel array unit 100 into digital information. The output signal supplied from the pixel array unit 100 is output in response to the timing at which light is received by a light receiving element included in the pixel 10 to which the pixel signal corresponds. The conversion unit 110 converts the supplied output signal into time information of a digital value indicating the timing. The output signals of all pixels 10 in the pixel array unit 100 are input to the conversion unit 110. The conversion unit 110 converts all output signals into time information and outputs it to the recognition unit 111.

[0036] The recognition unit 111 includes a DNN (Deep Neural Network) 111a, which is an example of a machine learning model. The DNN 111a is an algorithm with a multi-layer structure modeled on a human brain neural circuit (neural network) designed by machine learning to recognize time information corresponding to the distance D to the object to be measured (hereinafter, also referred to as time information of the object to be measured) from time information corresponding to the output signals of all the pixels 10 (hereinafter, also referred to as time information of each pixel 10). Note that the DNN 111a is an example, and the recognition unit 111 may use a model (learner) of any type, such as a regression model such as SVM, as the machine learning model.

[0037] The recognition unit 111 recognizes the time information of the object to be measured by inputting the time information of each pixel 10 input from the conversion unit 110 to the DNN 111a and executing DNN processing. Then, the recognition unit 111 outputs the DNN result output from the DNN 111a to the data generation unit 112 as a recognition result.

[0038] The data generation unit 112 generates output data from the DNN result input from the recognition unit 111, and outputs the output data to the storage device 7. Specifically, the data generation unit 112 calculates the distance D to the object to be measured based on the time information of the object to be measured recognized by the recognition unit 111, generates output data, and outputs the output data to the I / F unit 106.

[0039] The model switching unit 113 switches the machine learning model of the recognition unit 111. For example, based on an instruction from the control device 4, the model switching unit 113 reads out a model stored in the model storage unit 107 and supplies the model to the DNN 111a.

[0040] The output data generated by the data generation unit 112 is supplied to the I / F unit 106. The I / F unit 106 is, for example, a Mobile Industry Processor Interface (MIPI), and outputs the output data to, for example, the storage device 7. Alternatively, the I / F unit 106 may output the output data to the control device 4 or an external device (not shown).

[0041] (Layer structure of measuring device) Here, the laminated structure of the measuring device 1 will be briefly described with reference to Fig. 6. Fig. 6 is a schematic diagram showing an example of the laminated structure of the measuring device 1 according to an embodiment of the present disclosure. As shown in Fig. 6, the measuring device 1 has a first substrate P1 on which the pixel array section 100 is arranged, a second substrate P2 on which the conversion section 110 is arranged, and a third substrate P3 on which the recognition section 111 is arranged.

[0042] 6, the pixel array section 100 is disposed on a first substrate P1. Note that light receiving elements of the pixels 10 of the pixel array section 100 are disposed on the first substrate P1, and other circuit configurations included in the pixels 10 may be disposed on a substrate other than the first substrate P1, such as a second substrate P2.

[0043] The second substrate P2 is provided with a conversion unit 110. For example, when the conversion unit 110 has a time-to-digital conversion circuit 110a for each pixel 10, the time-to-digital conversion circuits corresponding to each pixel 10 are provided on the second substrate P2.

[0044] Here, the time-to-digital conversion circuit (TDC) 110a measures the time at which an output signal is supplied from the pixel 10, and converts the measured time into time information represented by a digital value. In this manner, the TDC 110a is a circuit that generates one piece of time information for one pixel 10, and the conversion unit 110 has, for example, the same number of TDCs 110a as the number of pixels 10.

[0045] The TDC 110a includes a counter that counts the time from the emission timing when the light source 2 emits light to the light reception timing when the pixel 10 receives the light, for example. The counter starts measuring (counting) time in synchronization with a light emission control signal supplied from the light emission control unit 105. The counter ends the time measurement in response to the inversion timing of the output signal supplied from the pixel 10. The TDC 110a outputs time information, which is the number of counts from when the counter starts to measure the time to when it ends, converted into a digital value, to the recognition unit 111.

[0046] 6, the first substrate P1 and the second substrate P2 are bonded to each other by, for example, so-called Cu-Cu bonding, which connects copper electrodes formed on the bonding surfaces of the substrates. Alternatively, the first substrate P1 and the second substrate P2 may be bonded to each other by, for example, flattening the bonding surfaces of the substrates and bonding them together by electron force, which is called direct bonding, or by other methods such as bump bonding.

[0047] The third substrate P3 has a recognition unit 111 disposed thereon. Note that logic circuits other than the recognition unit 111, such as a data generation unit 112 and an overall control unit 103, may also be disposed on the third substrate P3. Alternatively, logic circuits such as the data generation unit 112, model switching unit 113, and overall control unit 103 may be disposed on another substrate (not shown).

[0048] Time information is supplied from all the TDCs 110a to the recognition unit 111 disposed on the third substrate P3. The recognition unit 111 recognizes the time information of the object to be measured by using a machine learning model with the time information of each pixel 10 supplied from all the TDCs 110a as an input.

[0049] 6, the second substrate P2 and the third substrate P3 are bonded to each other by, for example, so-called Cu-Cu bonding, which connects copper electrodes formed on the bonding surfaces of the substrates. Alternatively, the second substrate P2 and the third substrate P3 may be bonded to each other by, for example, flattening the bonding surfaces of the substrates and bonding them together by electron force, which is called direct bonding, or by other methods such as bump bonding.

[0050] As described above, in the embodiment of the present disclosure, the pixel array section 100, the conversion section 110, and the recognition section 111 are disposed on different substrates (first to third substrates P1 to P3), and the measurement device 1 measures the light reception times of all the pixels 10 at once. This allows the recognition section 111 to recognize time information of the object to be measured based on the output signals of all the pixels 10. Therefore, the measurement device 1 can detect the distance D to the object to be measured without generating a histogram, and processing and memory for generating a histogram are not required.

[0051] In this embodiment, the pixel array section 100, the conversion section 110, and the recognition section 111 are arranged on different substrates (first to third substrates P1 to P3), but the present invention is not limited to this. For example, the pixel array section 100, the conversion section 110, and the recognition section 111 may be arranged on a single substrate.

[0052] Also, although an example has been shown here in which the output signals of all the pixels 10 are read out at once, the method of reading out the pixels 10 is not limited to this. For example, the output signal may be read out for each pixel 10 in a predetermined region. Alternatively, the output signal may be read out for each pixel 10, row or column. In this case, the measuring device 1 may have a memory that stores the output signal read out for each pixel 10 (or time information obtained by converting the output signal into time).

[0053] (Machine learning model) 7 to 9, a machine learning model used by the recognition unit 111 to recognize the distance D will be described. As described above, time information corresponding to the output signals of all the pixels 10 is input to the recognition unit 111. The recognition unit 111 inputs the time information of each pixel 10 to the DNN 11a and executes DNN processing, thereby recognizing the time information of the object to be measured.

[0054] An example of a neural network of a machine learning model will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining an overview of a neural network.

[0055] As shown in Fig. 7, the neural network 40 has a network structure consisting of three layers, an input layer 41, an intermediate layer 42, and an output layer 43, and the nodes included in each layer are connected to each other by links. In Fig. 7, the circles correspond to nodes, and the arrows correspond to links. When input data is input to the input layer 41, calculations are performed in the nodes and weighting is performed in the links in the order from the input layer 41 to the intermediate layer 42, and from the intermediate layer 42 to the output layer 43, and output data is output from the output layer 43. Note that among neural networks, those having a predetermined number of layers or more are also called DNNs (Deep Neural Networks) or deep learning.

[0056] Moreover, the neural network 40 shown in Fig. 7 is merely an example, and any network configuration may be used as long as it can realize the desired function. For example, in the example of Fig. 7, for simplicity of explanation, the output layer 43 has one node, but in the case of a classification model, the output layer 43 may have multiple nodes (for example, the number of classes to be classified).

[0057] It is known that neural networks can approximate any function. By using a calculation method such as backpropagation, neural networks can learn a network structure that matches the training data. Therefore, by constructing a model using a neural network, the model is freed from the constraint of expressive power, which is to be designed within the range that humans can understand.

[0058] The machine learning model used by the recognition unit 111 to recognize time information is not limited to the DNN 111a, and may be configured by various networks. For example, the machine learning model may be a model (learner) of any type, such as a regression model such as a Support Vector Machine (SVM). For example, the machine learning model may be various regression models, such as a nonlinear regression model or a linear regression model.

[0059] Moreover, the machine learning model used by the recognition unit 111 to recognize the time information is switched depending on the type of the object to be measured. This point will be described with reference to Fig. 8 and Fig. 9. Fig. 8 and Fig. 9 are diagrams for explaining examples of the machine learning model.

[0060] Light emitted from light source 2 is reflected by the object to be measured and received by measuring device 1. At this time, how much of the reflected light reaches measuring device 1 varies depending on the type of object to be measured.

[0061] 8, when the object to be measured is a car, most of the irradiated light is reflected by, for example, a body part formed of a flat surface and received by the measuring device 1. Therefore, the pixel 10 that receives the ambient light receives the light reception timing tc The number of pixels 10 receiving reflected light in the vicinity increases.

[0062] Therefore, as shown in the graph of FIG. 8, the distribution of the light receiving timing is c The distribution is such that the number of pixels that receive reflected light is large within a certain range including the bin. In the graph shown in Fig. 8, the vertical axis indicates the frequency for each bin, and the horizontal axis indicates the bin (time t), showing the relationship between the timing at which each pixel receives light and the number of pixels. In other words, the graph shown in Fig. 8 shows at what time there are many pixels that receive reflected light.

[0063] On the other hand, as shown in Fig. 9, when the object to be measured is a tree, a part of the irradiated light is diffused on the surface of the tree, so the reflected light that reaches the measuring device 1 is less than that in the case of a car. Therefore, as shown in Fig. 9, the distribution of the light receiving timing is a function of the light receiving timing t t However, compared to when the object to be measured is a car, the light reception timing t t The number of pixels receiving reflected light in the vicinity is reduced.

[0064] In this way, the distribution of the light reception timing changes not only depending on the distance D to the object to be measured, but also depending on the type of the object to be measured. Therefore, in the embodiment of the present disclosure, a machine learning model is trained for each type of object to be measured, such as a car, a building, a road, a pedestrian, or a tree. This can improve the recognition accuracy of the time information of the object to be measured.

[0065] The machine learning model is assumed to be constructed in advance based on supervised learning by associating the distribution of light receiving timing with the time information of the object to be measured for each type of object to be measured as teacher data. The constructed machine learning model is assumed to be stored in advance in the model storage unit 107. The recognition unit 111 reads out the machine learning model from the model storage unit 107 based on an instruction from the model switching unit 113, thereby switching the machine learning model used for recognizing the time information.

[0066] The construction of the machine learning model is not limited to each type of the object to be measured. For example, a machine learning model may be constructed for each scene in the subject space. Specifically, a machine learning model may be constructed for each time, such as day or night, for each weather, such as rainy or sunny, or for each location, such as a country, region, urban area, or mountainous area. For example, ambient light other than the reflected light received by each pixel 10 changes between day and night. In addition, in the case of rainy weather, the timing of receiving light by the pixel 10 changes due to reflection by raindrops. In addition, road conditions, signs, and surrounding environments differ depending on the country, region, urban area, or mountainous area. Therefore, by constructing a machine learning model for each scene in the subject space, the recognition accuracy of the time information of the object to be measured in the subject space can be improved. In addition, a machine learning model may be constructed for each combination of a scene and a type of object to be measured, such as when recognizing the distance D to a car in a daytime urban area.

[0067] (Distance recognition using machine learning models) Next, the recognition of time information of the object to be measured by the recognition unit 111 will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining the recognition of time information by the recognition unit 111. Hereinafter, the irradiation of irradiation light by the light source 2 will also be referred to as a shot.

[0068] The recognition unit 111 acquires time information obtained by converting an output signal from the pixel array unit 100 for each shot by the conversion unit 110. The recognition unit 111 inputs the time information acquired for each shot to the DNN 111a and acquires a recognition result of the time information corresponding to the distance D of the object to be measured.

[0069] For example, a case will be described in which the light source 2 irradiates irradiation light N times as shown in Fig. 10. In this case, the nth irradiation by the light source 2 is called the nth shot (n = 1 to N). The recognition unit 111 sequentially acquires time information of the distribution as shown in Fig. 10 corresponding to the 1st to Nth shots by the light source 2.

[0070] When the recognition unit 111 acquires the time information of the pixel 10 for each of the first to Nth shots, it sequentially inputs the information to the DNN 111a and recognizes the time information of the object to be measured. For example, in the example shown in FIG. 10, the recognition unit 111 acquires time information t1 corresponding to the first shot as a recognition result by the DNN 11a and outputs it to the data generation unit 112. Similarly, the recognition unit 111 acquires time information t n (n=2 to N) is acquired as a recognition result by the DNN 11a and output to the data generation unit 112.

[0071] The data generating unit 112 receives, for example, time information t1 to t2 of the object to be measured from the recognition unit 111 for each of the first to Nth shots. N When the time information t1 to t2 is sequentially acquired, the distance D to the object corresponding to each shot is calculated based on the above-mentioned formula (1). N is t in Eq. (1). em -t re Corresponds to.

[0072] In this way, the data generating section 112 may calculate the distance D to the object to be measured for each of the first to Nth shots, or the data generating section 112 may calculate the distance D to the object to be measured for each of multiple shots. In this case, the data generating section 112 may calculate the distance D by using an average value of time information of the object to be measured for the multiple shots.

[0073] Alternatively, the data generating unit 112 may determine whether or not the value of the time information of the object to be measured has converged, and may use the time information in the shot that has been determined to have converged to calculate the distance D. When the value of the time information has converged and the data generating unit 112 has calculated the distance D, the data generating unit 112 may determine that the recognition by the recognition unit 111 has ended, and may end the measurement of the distance of the object to be measured.

[0074] Note that, although the case where the distance D is calculated based on the light emitted by the light source 2 multiple times has been described here, for example, the measuring device 1 may finish measuring the distance D in one shot. That is, the recognition unit 111 may perform DNN processing on the time information of each pixel 10 acquired corresponding to one shot to recognize the time information of the object to be measured, and the data generation unit 112 may calculate the distance D, and then the measurement of the distance D by the measuring device 1 may be finished. In this way, by calculating the distance D using a machine learning model, the measuring device 1 can measure the distance D in as little as one shot, and the measurement time for the distance D can be shortened.

[0075] [Control device] Next, the control device 4 will be described in detail with reference to Fig. 11. The control device 4 controls the distance measurement system and also controls the switching of the machine learning model by the model switching unit 113. Here, the control device 4 mainly controls the switching of the machine learning model. Fig. 11 is a block diagram showing an example configuration of the control device 4 according to an embodiment of the present disclosure.

[0076] The control device 4 is realized by electronic circuits such as a CPU (Central Processing Unit), a microprocessor, etc. The control device 4 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, etc., and a RAM (Random Access Memory) that temporarily stores appropriately changing parameters, etc.

[0077] The control device 4 detects an object to be measured and recognizes a scene in the subject space based on an image captured by the imaging device 8, and controls switching of the machine learning model of the measuring device 1. Specifically, the control device 4 functions as an acquisition unit 401, an extraction unit 402, a scene recognition unit 403, a distance measurement determination unit 406, a model selection unit 404, and a notification unit 405.

[0078] The acquisition unit 401 acquires an RGB image from the imaging device 8. The acquisition unit 401 may acquire information regarding, for example, the imaging time and imaging location of the RGB image. Such information may be acquired from the imaging device 8, or may be acquired from a sensor such as a GPS sensor (not shown). Such a sensor may be included in the distance measuring system 6, or may be an external sensor.

[0079] The extraction unit 402 extracts the object to be measured from the RGB image. The extraction unit 402 performs image processing such as template matching, and extracts the object to be measured such as a tree, a vehicle, or a road from the RGB image. The extraction unit 402 may also extract a sky region, a ground region including a road, or the like from color information of the RGB image. Alternatively, the extraction unit 402 may extract the object to be measured, the sky region, the ground region, or the like using a machine learning model such as DNN.

[0080] The scene recognition unit 403 recognizes a scene in the subject space from the RGB image. Alternatively, the scene recognition unit 403 may recognize a scene from information related to the image capture time and image capture location of the RGB image. Furthermore, the scene recognition unit 403 may obtain information required for scene recognition from an external device or the like via a network (not shown).

[0081] Here, a scene is information indicating a subject space represented by surrounding information such as season, time, weather, or location. When recognizing a scene from an RGB image, the scene recognition unit 403 recognizes whether the subject space is daytime or nighttime, for example, from the brightness (luminosity) of the RGB image. The scene recognition unit 403 may also recognize a scene based on the extraction result by the extraction unit 402. For example, when the extraction unit 402 detects buildings, pedestrians, cars, etc. as objects to be measured, the scene recognition unit 403 recognizes that the subject space is an urban area.

[0082] Note that scene recognition by the scene recognition unit 403 is not limited to that based on RGB images, and scenes may be recognized from information other than RGB images. For example, the scene recognition unit 403 may recognize the season, time, etc. of the subject space from the image capture date and time of the RGB image. Also, the scene recognition unit 403 may recognize the location of the subject space from the image capture location of the RGB image. The scene recognition unit 403 may recognize the weather of the subject space based on, for example, the detection result of a rain detection sensor and the RGB image. In this way, the scene recognition unit 403 may recognize the scene of the subject space based on the detection results of multiple sensors, etc. Alternatively, it is also possible to acquire time information, weather, etc. from an external device via a network.

[0083] Alternatively, the scene recognition unit 403 may recognize the scene of the subject space using a machine learning model such as a DNN. In this case, the scene recognition unit 403 may input an RGB image to the DNN, or may input information such as the shooting date and time and the detection result of a rain detection sensor in addition to the RGB image to the DNN.

[0084] In this way, the scene recognition unit 403 recognizes the scene of the subject space based on the detection result of the sensor including, for example, an RGB image. For example, the scene recognition unit 403 recognizes that the subject space is “Japan”, “urban area”, “sunny”, and “evening”.

[0085] The ranging determination unit 406 determines a ranging position (ranging point) for measuring the distance D based on the extraction result of the extraction unit 402 and the recognition result of the scene recognition unit 403. The ranging determination unit 406 also determines the values ​​of various parameters related to the light emission system, such as the direction, power, and pulse shape of the irradiation by the light source 2. The ranging determination unit 406 also selects the values ​​of various parameters related to the light receiving system, such as the exposure period and frame rate of the pixel array unit 100.

[0086] 12 and 13, a description will be given of the distance measurement positions (distance measurement points) determined by distance measurement determination unit 406. Fig. 12 and Fig. 13 are diagrams for explaining the distance measurement positions determined by distance measurement determination unit 406.

[0087] 12, it is assumed that the extraction unit 402 extracts a tree Tr, a road R, a car C1, a house H, a pedestrian Pe, and a sky region SR from the RGB image M1. At this time, the distance measurement determination unit 406 determines to which object in the subject space the distance D should be measured, based on the extraction result of the extraction unit 402. The distance measurement determination unit 406 also determines at which position of the object the distance D should be measured.

[0088] Specifically, the distance measurement determination unit 406 determines, for example, to measure the distance D to the pedestrian Pe at five locations indicated by "+" in Fig. 12. The number N1 of distances D to be measured for each object is determined in advance for each type of object, for example, and the distance measurement determination unit 406 determines the distance measurement position in the RGB image M1, in other words, the direction of light irradiated from the light source 2, according to the number N1.

[0089] The distance measurement determination unit 406 sets the number of measurements N1 determined for each type of object to be measured to be the largest when the object to be measured is a moving object such as a pedestrian Pe or a car C1, and sets the number of measurements N1 to be smaller when the object to be measured is a stationary object such as a tree Tr or a house H (see FIG. 12). Also, the number of measurements N1 for a background such as a road R may be smaller than that for stationary objects. Also, it is not necessary to measure the distance D in an area without an object to be measured, such as the sky area SR. That is, since it is considered that there is no object to be measured in the sky area SR, the light source 2 does not need to irradiate light toward the sky area SR. Therefore, the distance measurement determination unit 406 determines to measure the distance D to the object to be measured excluding the sky area SR.

[0090] In this way, the distance measurement determination unit 406 determines the direction of light emitted from the light source 2 (distance measurement position) based on the type of the object extracted by the extraction unit 402 and its position in the RGB image.

[0091] Alternatively, as shown in Fig. 13, for example, when the extraction unit 402 extracts a car C2 facing sideways, the distance measurement determination unit 406 determines one position A1 as the measurement position for measuring the distance D to the car C2. This is because when the car C2 faces sideways, the side of the car C2 is composed of a substantially flat surface, so the distance D to the car C2 does not change significantly no matter where the distance is measured on the side. In this way, when the object to be measured has a flat surface, the distance measurement determination unit 406 determines a representative position of the flat surface as the distance measurement position. This allows the distance measurement system 6 to reduce the number of times the measuring device 1 measures the distance D to the object to be measured.

[0092] Although the distance measurement determination unit 406 determines one representative position here, the present invention is not limited to this, and the distance measurement determination unit 406 may determine a plurality of representative positions.

[0093] Furthermore, the distance measurement determination unit 406 may determine a distance measurement position according to the scene using the recognition result of the scene recognition unit 403. For example, when the time zone of the subject space is "night", the distance measurement determination unit 406 may increase the number N1 of measurements of the distance D to the object to be measured compared to when the time zone is "day". In other words, when the time zone is "night", the distance measurement determination unit 406 may determine the measurement positions such that the distance D to the object to be measured is measured at more measurement positions than when the time zone is "day".

[0094] Returning to Figure 11, the model selection unit 404 selects a machine learning model to be used for recognizing the distance D of the object to be measured by the measuring device 1 based on the extraction result of the extraction unit 402 and the recognition result of the scene recognition unit 403, for example, for each ranging position determined by the ranging determination unit 406.

[0095] Specifically, for example, when the extraction unit 402 extracts a pedestrian, the model selection unit 404 selects a machine learning model constructed by supervised learning using the pedestrian as correct answer data. In this way, the model selection unit 404 selects a machine learning model according to the type of the measured object extracted by the extraction unit 402, for example.

[0096] Alternatively, the model selection unit 404 may select a machine learning model appropriate for the scene using the recognition result of the scene recognition unit 403. For example, when the weather in the subject space is "rainy," the model selection unit 404 selects a machine learning model appropriate for the scene based on the scene recognition result, such as selecting a machine learning model constructed based on measurement data collected during rainy weather.

[0097] Furthermore, the model selection unit 404 may select a machine learning model based on, for example, the extraction result of the extraction unit 402 and the recognition result of the scene recognition unit 403. For example, a keyword related to the machine learning model is linked in advance, and the model selection unit 404 selects a machine learning model linked to the same keyword as the extraction result and the recognition result. Specifically, for example, when the extraction result is "car" and the scene recognition result is "daytime" and "rain", the model selection unit 404 checks the extraction result and the scene recognition result against the keywords linked to multiple machine learning models, and selects the machine learning model with the highest degree of match between the result and the keyword.

[0098] Alternatively, the model selection unit 404 may select a model using machine learning. In this case, the model selection unit 404 selects a machine learning model to be used for measuring the distance D by inputting the extraction result and the scene recognition result to a DNN, which is an example of a machine learning model, and executing a DNN process.

[0099] The notification unit 405 notifies the measurement device 1 of the selection result of the model selection unit 404. For example, the notification unit 405 notifies the measurement device 1 of information specifying the machine learning model selected by the model selection unit 404. Furthermore, when the model selection unit 404 selects parameter values ​​related to the light emission system and the light receiving system, the notification unit 405 notifies the measurement device 1 or the light source 2 of the selected parameter values.

[0100] [Distance measurement procedure] Next, a description will be given of an example of a distance measurement process procedure executed by the distance measurement system 6. First, a model selection process by the control device 4 will be described, and then a measurement process procedure by the measuring device 1 will be described.

[0101] (Model selection process) FIG. 14 is a flowchart illustrating an example of a procedure of a model selection process according to an embodiment of the present disclosure.

[0102] 14, the control device 4 acquires an RGB image from the imaging device 8 (step S201). Next, the control device 4 extracts a feature amount from the RGB image to extract the object to be measured (step S202).

[0103] Next, the control device 4 recognizes the scene of the RGB image (step S203). The control device 4 determines the distance measurement position of the distance D of the object based on the object extracted in step S202 and the scene recognized in step S203 (step S204).

[0104] The control device 4 selects a machine learning model for each ranging position (step S205), and notifies the measurement device 1 of the determined ranging position and the corresponding machine learning model (step S206).

[0105] (Measurement processing) FIG. 15 is a flowchart illustrating an example of a procedure of a measurement process according to an embodiment of the present disclosure.

[0106] 15, the measurement device 1 first switches the machine learning model used for recognition by the recognition unit 111 based on the notification from the control device 4 and the distance measurement position (step S101). Next, the measurement device 1 controls the light source 2 to emit light (step S102).

[0107] The measuring device 1 measures the time from when the light source 2 irradiates light to when the pixel 10 receives the light for all pixels 10 (step S103). The measuring device 1 recognizes time information of the object to be measured from the time measurement results for all pixels 10 using a machine learning model (step S104). The measuring device 1 generates data including the distance D to the object to be measured based on the recognized time information of the object to be measured (step S105).

[0108] As described above, the measurement device 1 according to the embodiment of the present disclosure includes a plurality of light receiving elements (pixels 10) and a recognition unit 111. The plurality of light receiving elements are disposed on a first substrate P1, and output a signal when light irradiated from a light source 2 and reflected by an object to be measured is received. The recognition unit 111 is disposed on a second substrate (corresponding to the third substrate P3 in the above embodiment) different from the first substrate P1, and recognizes a distance D to the object to be measured using a machine learning model based on output signals from the plurality of light receiving elements.

[0109] This allows the measuring device 1 to recognize time information of the object to be measured based on output signals from multiple light receiving elements, thereby reducing the processing time required to measure the distance D to the object to be measured. Furthermore, since the measuring device 1 does not generate a histogram and does not require a memory, it is possible to reduce the effort required for parameter adjustment compared to measuring the distance D using a histogram, and improve accuracy.

[0110] [Variations] In the above embodiment, the control device 4 selects a machine learning model based on an RGB image, but this is not limited to this. For example, the measurement device 1 may select a machine learning model based on time information of all pixels 10.

[0111] For example, assume that the distribution of the time information of pixels 10 is not dense in a specific region but is a random distribution, as shown in Fig. 16. In this case, the model switching unit 113 of the measurement device 1 selects a machine learning model according to such distribution and switches the machine learning model used for recognition by the recognition unit 111. The model switching unit 113 may obtain the distribution of the time information of pixels 10 from, for example, the conversion unit 110, or may obtain it from the recognition unit 111. Note that Fig. 16 is a diagram showing an example of the distribution of the time information of pixels 10.

[0112] Alternatively, for example, the model switching unit 113 of the measurement device 1 may change the values ​​of various parameters related to the light emission system, such as the direction, power, and pulse shape of irradiation by the light source 2, based on such distribution. For example, when the time information of the pixel 10 is randomly distributed as shown in Fig. 16, the model switching unit 113 changes the values ​​of various parameters so as to determine that the distance D to the object to be measured is far and increase the irradiation power, or to determine that there is no object to be measured in the irradiation direction and change the irradiation direction. The model switching unit 113 outputs the changed parameters to, for example, the light emission control unit 105. Alternatively, the light emission control unit 105 may change the various parameters based on such distribution.

[0113] Also, assume that the distribution of the time information of the pixels 10 is concentrated in a specific region as shown in Fig. 17. In other words, when the detection result of the light reception timing by the pixels 10 is a saturated distribution, the model switching unit 113 of the measurement device 1 selects a machine learning model according to such distribution and switches the machine learning model used for recognition by the recognition unit 111. Note that Fig. 17 is a diagram showing another example of the distribution of the time information of the pixels 10.

[0114] Alternatively, for example, the model switching unit 113 of the measurement device 1 may change the values ​​of various parameters related to the light emission system, such as the direction, power, and pulse shape of irradiation by the light source 2, based on such distribution. The model switching unit 113 outputs the changed parameters to, for example, the light emission control unit 105. Alternatively, the light emission control unit 105 may change the various parameters based on such distribution.

[0115] Although the measurement device 1 changes various parameters of the light-emitting system here, it may also change various parameters of the light-receiving system, such as the light-receiving sensitivity.

[0116] In this way, the measuring device 1 can improve the measurement accuracy of the distance D to the object to be measured by switching the machine learning model or changing various parameters of the light-emitting system or the light-receiving system based on the time information of the pixel 10.

[0117] In the above embodiment, the recognition unit 111 of the measuring device 1 recognizes the time information of the object to be measured, but this is not limited to the above. The recognition unit 111 of the measuring device 1 may input the time information of each pixel 10 to the DNN 111a to recognize the distance D to the object to be measured. In this way, the recognition unit 111 recognizes information related to the distance D to the object to be measured, such as the time information and distance D of the object to be measured.

[0118] In the above embodiment, the control device 4 selects a machine learning model based on the imaging result of the imaging device 8, but the control device 4 may select a machine learning model based on the sensing result of another sensor device.

[0119] For example, the control device 4 may select a machine learning model based on the sensing result of an infrared sensor such as a near-infrared (IR) sensor, a short-wave infrared (SWIR) sensor, a mid-wave infrared (MWIR) sensor, or a long-wave infrared (LWIR) sensor instead of the imaging device 8. Alternatively, the control device 4 may select a machine learning model based on the sensing result of a LIDAR (Laser Imaging Detection and Ranging) sensor that measures the distance D based on light of a frequency band different from the frequency band of light irradiated from the light source 2. The control device 4 may also select a machine learning model based on the sensing result of a RADAR (Radio Detecting and Ranging) sensor.

[0120] Alternatively, the control device 4 may combine the sensing results of various sensors including the imaging device 8 described above to select a machine learning model.

[0121] In addition, the recognition unit 111 may recognize the time information of the object to be measured using the sensing results of the various sensors described above. In this case, the outputs of the various sensors described above are input to the recognition unit 111. The recognition unit 111 recognizes the time information of the object to be measured by inputting the time information of each pixel 10 and the outputs of the various sensors described above to the DNN 111a and executing DNN processing. In this way, by using the outputs of the various sensors as well, the recognition accuracy of the recognition unit 111 in recognizing the time information of the object to be measured can be improved.

[0122] In the above embodiment, the measuring device 1 measures the distance D to the object to be measured by switching the machine learning model, but this is not limited to the above. For example, when one machine learning model can handle multiple types of objects to be measured, the measuring device 1 may measure the distance D to the object to be measured using one machine learning model. Alternatively, when the measuring device 1 measures the distance D to a specific object to be measured, such as a car, the measuring device 1 may measure the distance D using one machine learning model specialized for cars.

[0123] Also, for example, in the embodiment, an example has been shown in which the data generation unit 112 and the model switching unit 113 are provided inside the measurement device 1, but the data generation unit 112 and the model switching unit 113 may also be provided in an application processor provided outside the distance measuring device.

[0124] Alternatively, for example, in the embodiment, an example has been shown in which the distance measurement determination unit 406 and the model selection unit 404 are provided inside the control device 4, but the distance measurement determination unit 406 and the model selection unit 404 may be provided in the measurement device 1. In this case, the model switching unit 113 may be provided with the function of the model selection unit 404, so that the model selection unit 404 is provided in the measurement device 1.

[0125] [effect] The measuring device 1 in the embodiment and the modified example includes a plurality of light receiving elements (pixels 10) arranged on a first substrate P1, which output a signal when light irradiated from a light source 2 and reflected by the object to be measured is received, and a recognition unit 111 arranged on a third substrate P3 different from the first substrate P1, which recognizes information regarding the distance D to the object to be measured using a machine learning model (DNN 111a) based on the output signals of the plurality of light receiving elements.

[0126] This allows the measurement device 1 to measure the distance D to the object to be measured without generating a histogram, and the processing time required to measure the distance D can be reduced.

[0127] Moreover, the measurement device 1 according to the embodiment and the modified example further includes a time measurement unit (conversion unit 110) that measures the time from the emission timing of the light source 2 to the reception timing of the light by the light receiving elements based on the output signals of the multiple light receiving elements and acquires the measurement value. The recognition unit 111 recognizes information about the distance D using a machine learning model that receives as input multiple measurement values ​​corresponding to the multiple light receiving elements.

[0128] This allows the measurement device 1 to measure the distance D to the object to be measured without generating a histogram, and the processing time required to measure the distance D can be reduced.

[0129] Moreover, the recognition unit 111 according to the embodiment and the modified example recognizes information about the distance D based on the measurement value measured by the time measurement unit (conversion unit 110) when the light source 2 emits light once.

[0130] In this way, the recognition unit 111 recognizes information about the distance D with one light emission from the light source 2, and the measurement device 1 can reduce the measurement time for the distance D.

[0131] Furthermore, when the light source 2 emits light multiple times, the recognition unit 111 according to the embodiment and the modified example recognizes information about the distance D based on the measurement values ​​measured by the time measurement unit (conversion unit 110) corresponding to each light emission.

[0132] In this way, the recognition unit 111 recognizes information about the distance D by multiple light emissions from the light source 2, and the measurement device 1 can improve the measurement accuracy of the distance D.

[0133] Furthermore, when it is determined that the recognition unit 111 according to the embodiment and the modified example has recognized the information related to the distance D, the recognition unit 111 ends the recognition of the information related to the distance D.

[0134] In this way, by ending the recognition process when the recognition unit 111 recognizes information about the distance D, the measurement device 1 can improve the measurement accuracy of the distance D.

[0135] Moreover, the recognition unit 111 according to the embodiment and the modified example recognizes information about the distance D by using a neural network as a machine learning model.

[0136] This enables the measurement device 1 to improve the measurement accuracy of the distance D.

[0137] Moreover, the connection between the first substrate P1 and the third substrate P3 according to the embodiment and the modified example is a connection between copper electrodes.

[0138] This improves the degree of freedom in designing the measuring device 1, and also enables the measuring device 1 to be made more compact.

[0139] In addition, the distance measuring device of the embodiment and the modified example includes a light source 2 that irradiates light onto the object to be measured, a plurality of light receiving elements (pixels 10) arranged on a first substrate P1 and that output a signal when it receives light reflected from the object to be measured, and a recognition unit 111 arranged on a third substrate P3 different from the first substrate P1 and that recognizes information regarding the distance D to the object to be measured using a machine learning model (DNN 111a) based on the output signals of the plurality of light receiving elements.

[0140] This allows the distance measuring device to measure the distance D to the object without generating a histogram, and the processing time required to measure the distance D can be reduced.

[0141] [Application examples of the technology disclosed herein] Next, application examples of the embodiments and modifications of the present disclosure will be described. Fig. 18 is a diagram showing a usage example of a measurement device 1 to which the above-described embodiments and modifications can be applied.

[0142] The above-described measuring device 1 can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as described below.

[0143] -Digital cameras, mobile devices with camera functions, and other devices that take images for viewing. Devices used for traffic purposes, such as on-board sensors that photograph the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognition of the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliance in accordance with those gestures. - Equipment used for medical or healthcare purposes, such as endoscopes and devices that take blood vessel images by receiving infrared light. - Equipment used for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. - Equipment used for beauty purposes, such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp. Equipment used for sports, such as action cameras and wearable cameras for sports purposes. Agricultural equipment, such as cameras for monitoring fields and crop conditions.

[0144] (Example of application to moving objects) The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, or a robot.

[0145] FIG. 19 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0146] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 19, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053 are illustrated.

[0147] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0148] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0149] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0150] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0151] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0152] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0153] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0154] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0155] The audio / image output unit 12052 transmits at least one output signal of audio and image to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 19, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0156] FIG. 20 is a diagram showing an example of the installation position of the imaging unit 12031.

[0157] In FIG. 20, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0158] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield in the vehicle interior mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided at the upper part of the windshield in the vehicle interior is mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0159] 20 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0160] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0161] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0162] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0163] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0164] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 among the configurations described above. Specifically, the distance measuring system 6 in FIG. 4 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to improve the calculation speed of the distance to an obstacle around the vehicle.

[0165] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure. In addition, components of different embodiments and modifications may be appropriately combined.

[0166] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0167] The present technology can also be configured as follows. (1) a plurality of light receiving elements disposed on a first substrate, the light receiving elements outputting a signal when receiving light irradiated from a light source and reflected by an object to be measured; a recognition unit that is disposed on a second substrate different from the first substrate and recognizes information regarding a distance to the object to be measured using a machine learning model based on output signals of the plurality of light receiving elements; A measuring device comprising: (2) a time measurement unit that measures a time from a light emission timing at which a light source emits light to a light reception timing at which the light receiving elements receive light based on the output signals of the plurality of light receiving elements and acquires a measurement value; The recognition unit recognizes information related to the distance using the machine learning model to which a plurality of the measurement values ​​corresponding to the plurality of light receiving elements are input. A measuring device as described in (1). (3) The measurement device according to (2), wherein the recognition unit recognizes information regarding the distance based on the measurement value measured by the time measurement unit when the light source emits light once. (4) The measurement device according to (2), wherein the recognition unit recognizes information regarding the distance based on the measurement value measured by the time measurement unit corresponding to each light emission when the light source emits light multiple times. (5) The measuring device according to (4), wherein when it is determined that the information relating to the distance has been recognized, the recognition unit terminates recognition of the information relating to the distance. (6) The measurement device according to any one of (1) to (5), wherein the recognition unit recognizes the information relating to the distance by using a neural network as the machine learning model. (7) The measuring device according to any one of (1) to (6), wherein the connection between the first substrate and the second substrate is a connection between copper electrodes. (8) A light source that irradiates light onto the object to be measured; a plurality of light receiving elements disposed on a first substrate, the light receiving elements outputting signals when receiving light reflected from the object to be measured; a recognition unit that is disposed on a second substrate different from the first substrate and recognizes information regarding a distance to the object to be measured using a machine learning model based on output signals of the plurality of light receiving elements; A distance measuring device comprising: [Explanation of symbols]

[0168] 1. Measuring equipment 2 light source 4. Control device 6. Ranging System 8. Imaging device 10 pixels 100 Pixel array section 107 Model Memory Section 110 Conversion unit 111 Recognition part 112 Data Generation Unit 113 Model Switching Unit 401 Acquisition Department 402 Extraction part 403 Scene Recognition Unit 404 Model Selection Section 405 Notification Department 406 Distance determination unit

Claims

1. a plurality of light receiving elements disposed on a first substrate, the light receiving elements outputting signals when receiving light irradiated from a light source and reflected by an object to be measured; a time measurement unit that measures a time from a light emission timing at which the light source emits light to a light reception timing at which the light receiving elements receive light based on output signals from the plurality of light receiving elements, and measures a time measurement value; a calculation unit that is disposed on a second substrate different from the first substrate and that calculates a distance to the object based on the output signals of the light receiving elements by using a machine learning model that receives as input a plurality of the time measurement values ​​corresponding to each of the light receiving elements; A measuring device comprising:

2. The measuring device according to claim 1 , wherein the calculation section calculates the distance based on the time measurement value measured by the time measurement section when the light source emits light once.

3. The measurement device according to claim 1 , wherein the calculation section calculates the distance based on the time measurement value measured by the time measurement section in response to each light emission when the light source emits light a plurality of times.

4. The measuring device according to claim 3 , wherein the calculation section terminates the calculation of the distance when it is determined that the information relating to the distance has been recognized.

5. The measurement device according to claim 4 , wherein the calculation unit calculates the distance using a neural network as the machine learning model.

6. 6. The measuring device according to claim 5, wherein the connection between the first substrate and the second substrate is a connection between copper electrodes.

7. The measuring device as described in claim 1, further comprising a decision unit that selects the machine learning model and / or determines parameters for obtaining the time measurement value based on information about the subject space in which the object to be measured is located.

8. A light source that irradiates light onto the object to be measured; a plurality of light receiving elements disposed on a first substrate, the light receiving elements outputting signals when receiving light reflected from the object to be measured; a time measurement unit that measures a time from a light emission timing at which the light source emits light to a light reception timing at which the light receiving elements receive light based on output signals from the plurality of light receiving elements, and measures a time measurement value; a calculation unit that is disposed on a second substrate different from the first substrate and that calculates a distance to the object based on the output signals of the light receiving elements by using a machine learning model that receives as input a plurality of the time measurement values ​​corresponding to each of the light receiving elements; A distance measuring device comprising:

Images