Depth information processing device, depth distribution estimation method, depth distribution detection system, and trained model generation method

Abstract

To reduce the distortion and displacement of a depth distribution attributable to a variation in the measurement time of a distance measuring device.SOLUTION: A depth information processing device in an embodiment of the present disclosure comprises a distance information acquisition unit, an image acquisition unit, and a processor. The distance information acquisition unit acquires output data from a scanning type distance measuring device. The image acquisition unit acquires an image in which a range overlapping the detection range of the distance measuring device is imaged. The processor estimates, on the basis of the output data and image, a depth distribution at a prescribed timing in a first period which is the one frame period of the distance measuring device. In order to estimate the depth distribution, the processor performs a plurality of processing to generate composite depth information from the output data, for each of a plurality of second periods into which the first period is divided in time. The processor inputs the generated composite depth information to a trained model so as to estimate a depth distribution.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a depth information processing device, a depth distribution estimation method, a depth distribution detection system, and a trained model generation method. [Background technology]

[0002] A scanning distance measuring device such as a laser radar is used to measure the direction and distance to an object. For example, a technology has been disclosed in which a laser radar is used to measure the distance and direction to an object around the vehicle, and the location of a parked vehicle is determined based on the acquired data point sequence (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-243857 Summary of the Invention [Problem to be solved by the invention]

[0004] However, because conventional laser radars perform sequential scanning using lasers, the resolution and frame rate are lower than those of camera images. Furthermore, in scanning laser radars, increasing the number of measurement points to increase resolution further increases the scanning time. Furthermore, if the detection target is a moving object, the object will move during scanning. Therefore, differences in measurement time between measurements in one frame can cause distortions and deviations in the depth distribution, making accurate measurements impossible.

[0005] Therefore, the purpose of the present disclosure, which has been made with these points in mind, is to reduce distortion and deviation in the depth distribution caused by differences in measurement time when measuring the depth distribution using a scanning distance measuring device. [Means for solving the problem]

[0006] A depth information processing device according to an embodiment of the present disclosure includes a distance information acquisition unit, an image acquisition unit, and a processor. The distance information acquisition unit acquires output data from a scanning distance measurement device. The image acquisition unit acquires an image of a range overlapping with the detection range of the distance measurement device. The processor estimates a depth distribution at a predetermined timing within a first period, which is the period of one frame of the distance measurement device, based on the output data and the image. The processor generates, from the output data, a plurality of pieces of depth information for each of a plurality of second periods obtained by temporally dividing the first period, the plurality of pieces including depth, a two-dimensional position at which the depth was detected, and time information corresponding to the second period. The processor generates, from the image, region division information for dividing the detection range into a background region and a moving object region for each of the second periods. The processor generates, for each of the second periods, synthesized depth information, which includes accumulated depth information obtained by accumulating the plurality of depth information during the first period in the background region and includes the plurality of depth information for the second period in the moving object region. The processor further estimates the depth distribution by inputting the generated synthesized depth information to a trained model trained using training data that receives the plurality of synthesized depth information for the second periods as input and outputs a depth distribution at the predetermined timing within the first period.

[0007] A depth distribution estimation method according to an embodiment of the present disclosure is a method executed by a computer processor. The method includes acquiring output data from a scanning distance measuring device and acquiring an image capturing an area overlapping with a detection range of the distance measuring device. The method includes generating, from the output data, a plurality of pieces of depth information for each of a plurality of second periods obtained by temporally dividing a first period, which is the period of one frame of the distance measuring device, the plurality of pieces of depth information including depths, two-dimensional positions at which the depths were detected, and time information corresponding to the second period. The method also includes generating, from the image, region division information for each of the second periods that divides the detection range into a background region and a moving object region. The method also includes generating, for each of the second periods, synthesized depth information, in which the background region includes accumulated depth information obtained by accumulating the plurality of pieces of depth information over the first period and the moving object region includes the plurality of pieces of depth information over the second period. The method includes estimating the depth distribution by inputting the generated synthetic depth information into a trained model trained using training data that takes as input synthetic depth information for multiple second periods and outputs a depth distribution at a predetermined timing within the first period.

[0008] A depth distribution detection system according to an embodiment of the present disclosure includes a scanning distance measuring device, an imaging device, and and a depth information processing device. The imaging device captures an image of a range overlapping with the detection range of the ranging device. The depth information processing device includes a processor that calculates a depth distribution at a predetermined timing within a first period, which is the period of one frame of the ranging device, based on output data from the ranging device and an image captured by the imaging device. The processor generates, from the output data, a plurality of pieces of depth information for each of a plurality of second periods obtained by temporally dividing the first period, the pieces of depth information including depths, positions in two-dimensional directions at which the depths were detected, and time information corresponding to the second period. The processor generates, from the image, region division information that divides the detection range into a background region and a moving object region for each of the second periods. The processor generates, for each of the second periods, synthesized depth information that includes accumulated depth information obtained by accumulating the plurality of pieces of depth information for the first period in the background region and the plurality of pieces of depth information for the second period in the moving object region. The processor estimates the depth distribution by inputting the generated synthetic depth information into a trained model trained using training data that receives synthetic depth information for multiple second periods and outputs a depth distribution at the specified timing within the first period.

[0009] A trained model generation method according to an embodiment of the present disclosure is a computer-executed trained model generation method. The method acquires or generates a plurality of pieces of data combining depth information and region division information generated for each of a plurality of second periods obtained by temporally dividing a first period, which is the period of one frame of a scanning range finding device, and a depth distribution at a predetermined timing within the first period. Each piece of depth information includes a depth, a two-dimensional position at which the depth is detected, and time information corresponding to the second period, and the region division information divides the detection range of the range finding device into a background region and a moving object region. The method generates, for each of the second periods, synthesized depth information, in which the background region includes accumulated depth information obtained by accumulating the plurality of depth information over the first period, and the moving object region includes the plurality of depth information for the second period. The method generates a trained model by performing machine learning using training data in which the synthesized depth information for the plurality of second periods is input and the depth distribution is output. [Effects of the Invention]

[0010] According to an embodiment of the present disclosure, in measuring a depth distribution using a scanning distance measuring device, distortion and deviation of the depth distribution caused by deviation in measurement time can be reduced. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a configuration diagram showing a basic configuration of a depth distribution detection system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing a more detailed example of the configuration of the depth distribution detection system of FIG. 1. [Figure 3] FIG. 2 is a block diagram showing a schematic configuration of the depth information processing device of FIG. [Figure 4] 2 is a flowchart showing a process executed by the depth information processing device of FIG. 1. [Figure 5] 2 is a diagram showing an example of a scene detected by the distance measuring device and the imaging device of FIG. 1. FIG. [Figure 6]2 is a diagram illustrating an image and depth information acquired by a depth information processing device during a distance measurement period of one frame of the distance measuring device in FIG. 1. FIG. [Figure 7] FIG. 10 is a diagram illustrating a method for generating area division information. [Figure 8] FIG. 10 is a diagram illustrating a method for generating composite depth information. [Figure 9] FIG. 1 is a configuration diagram showing a schematic configuration of a training computer that generates a trained model. [Figure 10] 1 is a flowchart showing a process for generating a trained model by a training computer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The drawings used in the following description are schematic. The dimensions and ratios in the drawings do not necessarily correspond to the actual dimensions and ratios.

[0013] 1, a depth distribution detection system 10 according to an embodiment of the present disclosure includes a distance measuring device 11, an imaging device 12, and a depth information processing device 13. In this application, depth means the distance from the distance measuring device 11 to an object. In this application, depth is used synonymously with distance.

[0014] The distance measuring device 11 is a scanning type distance measuring device such as a laser radar. A laser radar irradiates a detection range with laser light and measures the time it takes for the reflected laser light to return. A laser radar is also called a LiDAR (Light Detection and Ranging). The imaging device 12 has an imaging optical system and an imaging element, and acquires an image of the detection range. The imaging device 12 is configured to capture an image of a range that overlaps with the detection range of the distance measuring device 11.

[0015] The distance measuring device 11 and the imaging device 12 can be configured so that the coordinate systems for observing the object are substantially the same or close to each other. Therefore, the distance measuring device 11 and the imaging device 12 may be configured so that the optical axis ax of the optical system 14 for detecting or imaging the object is the same. The optical system 14 may have various configurations. Furthermore, the distance measuring device 11 and the imaging device 12 may be arranged so that the optical axes of the optical systems for detecting or imaging the object are parallel and close to each other. In this case, the optical system 14 for aligning the optical axes of the distance measuring device 11 and the imaging device 12 may not be necessary. There is little or no parallax between the detected data and images of the distance measuring device 11 and the imaging device 12. The depth output by the distance measuring device 11 and the image output by the imaging device 12 can be superimposed so that their positions correspond.

[0016] The distance measuring device 11 and the image capturing device 12 are disposed fixedly relative to the background. Therefore, the distance measuring device 11 and the image capturing device 12 may be fixed to a stationary object. For example, when the depth distribution detection system 10 is used outdoors, the distance measuring device 11 and the image capturing device 12 may be fixed to a structure or the like provided on the roadside or above the road.

[0017] The depth information processing device 13 is a computer. The depth information processing device 13 may be any of various computers including a general-purpose computer, a workstation, a PC (Personal Computer), etc. The depth information processing device 13 may load dedicated programs and data to execute each function of the depth information processing device 13 described below.

[0018] The output data from the distance measuring device 11 acquired by the depth information processing device 13 includes at least two-dimensional position information indicating the direction of the measurement target and distance information. The two-dimensional position information corresponds to a position on an image captured by the imaging device 12. The output data may further include information on the time when the depth was measured. The depth information processing device 13 further acquires from the imaging device 12 the image captured by the imaging device 12.

[0019] The depth information processing device 13 estimates a depth distribution at a predetermined timing within a first period T1, which is the period of one frame of the distance measuring device 11, based on the output data output from the distance measuring device 11 and the image output from the imaging device 12. In this application, the "depth distribution" refers to a two-dimensional depth distribution when the detection range observed by the distance measuring device 11 is viewed in a planar view. The "depth distribution" is used in a similar sense to the "depth map." The "depth distribution" does not need to be expressed as an image. The predetermined timing can be, for example, the center time of one frame of the distance measuring device 11. The depth distribution estimated by the depth information processing device 13 has reduced distortion and deviation due to differences in measurement times of the distance measuring device 11. The depth information processing device 13 is equipped with a trained model based on machine learning, and can estimate the depth distribution through machine learning using the output data output from the distance measuring device 11 and the image output from the imaging device 12.

[0020] A more specific example of the depth distribution detection system 10 will be described with reference to Fig. 2. The depth distribution detection system 10 includes a detection device 21 that incorporates the functions of the distance measuring device 11 and the image capturing device 12, and a depth information processing device 13. The detection device 21 includes an irradiation unit 22, a reflecting unit 23, a control unit 24, a first optical system 25, and a detection element 26 as components corresponding to the distance measuring device 11. The detection device 21 also includes a control unit 24, a second optical system 27, and an image capturing element 28 as components corresponding to the image capturing device 12. The detection device 21 further includes a switching unit 32.

[0021] (Acquisition of depth information) The irradiation unit 22 emits at least one of infrared, visible light, ultraviolet, and radio waves. In one embodiment, the irradiation unit 22 emits infrared rays. The irradiation unit 22 irradiates the electromagnetic waves toward the object ob directly or indirectly via the reflector 23. In the embodiment shown in FIG. 2, the irradiation unit 22 irradiates the electromagnetic waves toward the object ob indirectly via the reflector 23.

[0022] In one embodiment, the irradiation unit 22 emits electromagnetic waves in the form of a narrow beam, for example, 0.5°. In another embodiment, the irradiation unit 22 can emit electromagnetic waves in pulses. For example, the irradiation unit 22 includes an LED (Light Emitting Diode) and an LD (Laser Diode). The irradiation unit 22 switches between emitting and stopping the electromagnetic waves under the control of the control unit 24, which will be described later.

[0023] The reflecting unit 23 reflects the electromagnetic waves emitted from the irradiating unit 22 while changing their direction, thereby changing the irradiation position of the electromagnetic waves irradiated onto the object ob. The irradiation position is equal to the detection position where the distance measuring device 11 measures the distance. The reflecting unit 23 scans the object ob with the electromagnetic waves emitted from the irradiating unit 22. Therefore, the detecting element 26 cooperates with the reflecting unit 23 to form a scanning distance measuring sensor.

[0024] The reflecting unit 23 can scan the electromagnetic wave irradiation position in two dimensions within the detection range. The reflecting unit 23 can perform regular scanning by scanning the irradiation position where the distance measuring device 11 performs detection in the horizontal direction while sequentially shifting it in the vertical direction, as in a progressive scanning method. The reflecting unit 23 may also sequentially set the electromagnetic wave irradiation position to a random position in two dimensions. In the present disclosure, "scanning" also includes performing sequential measurements by randomly changing the irradiation position. The reflecting unit 23 may scan the detection range using any other method.

[0025] The reflecting unit 23 is configured so that at least a part of the irradiation area of ​​the electromagnetic waves emitted from the irradiating unit 22 and reflected therefrom is included in the detection range of the detecting element 26. Therefore, at least a part of the electromagnetic waves irradiated to the object ob via the reflecting unit 23 can be detected by the detecting element 26.

[0026] The reflecting unit 23 includes, for example, a MEMS (Micro Electro Mechanical Systems) mirror, a polygon mirror, and a galvanometer mirror.

[0027] The reflecting unit 23 changes the direction in which it reflects the electromagnetic waves based on the control of the control unit 24, which will be described later. The reflecting unit 23 may also have an angle sensor, such as an encoder, and may notify the control unit 24 of the angle detected by the angle sensor as direction information in which it reflects the electromagnetic waves. In such a configuration, the control unit 24 can calculate the irradiation position based on the direction information acquired from the reflecting unit 23. The control unit 24 can also calculate the irradiation position based on a drive signal input to the reflecting unit 23 to cause it to change the direction in which it reflects the electromagnetic waves.

[0028] When the control unit 24 calculates the irradiation position, i.e., the detection position for detecting depth, based on the direction information acquired from the reflecting unit 23 or the drive signal input to the reflecting unit 23, the reflecting unit 23 can be disposed so that the central axis of the direction in which the electromagnetic wave is scanned is substantially parallel to and close to the optical axis ax. In this case, the detection axis of the distance measuring device 11 becomes the central axis of the electromagnetic wave scanned by the reflecting unit 23.

[0029] The control unit 24 includes one or more processors and memory. The processor may include at least one of a general-purpose processor that loads a specific program to execute a specific function and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 24 may include at least one of a system-on-a-chip (SoC) and a system in a package (SiP) in which one or more processors work together.

[0030] The control unit 24 is configured to be able to control the reflecting unit 23 and the switching unit 32. The control unit 24 can control the switching unit 32 so that the detecting element 26 can acquire the reflected electromagnetic wave, depending on the position and time of irradiation of the electromagnetic wave by the reflecting unit 23. The control unit 24 can acquire detection information from the detecting element 26 and generate depth information, as will be described later. The control unit 24 can also acquire an image signal from the imaging element 28. The control unit 24 can output the depth information and the image signal to the depth information processing device 13.

[0031] The first optical system 25 transmits the electromagnetic waves reflected from the object ob, which are irradiated from the irradiation unit 22 and reflected by the reflection unit 23 toward the detection range, so that the reflected waves can be detected by the detection element 26.

[0032] The detection element 26 includes an element capable of detecting electromagnetic waves emitted from the irradiation unit 22. For example, the detection element 26 includes a single element such as an APD (Avalanche PhotoDiode), a PD (PhotoDiode), or a ranging image sensor. The detection element 26 may also include an element array such as an APD array, a PD array, a ranging imaging array, or a ranging image sensor. In one embodiment, the detection element 26 transmits detection information indicating that it has detected reflected waves from the subject as a signal to the control unit 24. The detection element 26 detects, for example, electromagnetic waves in the infrared band.

[0033] In addition, in the configuration where the detection element 26 is a single element constituting the distance measuring sensor described above, it is sufficient that the detection element 26 can detect electromagnetic waves, and it is not necessary that the object ob be imaged on the detection surface. Therefore, the detection element 26 does not need to be provided at the secondary imaging position, which is the imaging position by the first post-stage optical system 30. In other words, in this configuration, the detection element 26 may be located anywhere on the path of the electromagnetic waves that travel in the first direction d1 by the switching unit 32 and then travel via the first post-stage optical system 30, as long as the detection element 26 is located at a position where electromagnetic waves from all angles of view can be incident on the detection surface.

[0034] The control unit 24 acquires the depth based on the electromagnetic waves detected by the detection element 26. The control unit 24 acquires the depth of the irradiation position irradiated by the irradiation unit 22 using a ToF (Time-of-Flight) method based on the detection information detected by the detection element 26, as will be described below.

[0035] The control unit 24 inputs an electromagnetic wave emission signal to the irradiation unit 22, causing the irradiation unit 22 to emit pulsed electromagnetic waves. The irradiation unit 22 irradiates electromagnetic waves based on the input electromagnetic wave emission signal. The electromagnetic waves emitted by the irradiation unit 22 and reflected by the reflection unit 23 and irradiated onto a given irradiation area are reflected from the irradiation area. When the detection element 26 detects the electromagnetic waves reflected from the irradiation area, it notifies the control unit 24 of the detection information.

[0036] The control unit 24 measures the time from when the irradiation unit 22 emits electromagnetic waves to when the detection information is acquired. The control unit 24 calculates the distance to the irradiation position by multiplying this time by the speed of light and dividing by two. If the detection element 26 is a single element, the control unit 24 calculates the irradiation position based on the directional information acquired from the reflector 23 or the drive signal output by the detection element 26 to the reflector 23, as described above. If the detection element 26 includes an array of elements, the control unit 24 can calculate the irradiation position based on the position on the element array where the electromagnetic waves reflected by the object ob are detected. In this case, the irradiation position can be identified using approximately the same coordinate axes as the image obtained from the imaging element 28, with no parallax. The control unit 24 generates output data including information on the depth and detection position by calculating the distance to each irradiation position while changing the irradiation position.

[0037] (Getting image information) The second optical system 27 forms an image of the object ob in the detection range of the detection element 26, that is, in the range overlapping with the detection range of the distance measuring device 11, on the detection surface of the imaging element .

[0038] The imaging element 28 converts the image formed on the detection surface into an electrical signal to generate an image of the detection range including the object ob. The imaging element 28 may include either a CCD image sensor (Charge-Coupled Device Image Sensor) or a CMOS image sensor (Complementary MOS Image Sensor).

[0039] The image sensor 28 outputs the generated image to the control unit 24. The control unit 24 may perform any processing on the image, such as distortion correction, brightness adjustment, contrast adjustment, gamma correction, etc. If there is a discrepancy between the coordinate system of the output data output from the detection element 26 and the coordinate system of the image output from the image sensor 28, the control unit 24 may adjust them so that they are closer to each other.

[0040] (Configuration of optical system and switching unit) The first optical system 25 includes a front optical system 29 shared with the second optical system 27, and a first rear optical system 30 located after the switching unit 32. The second optical system 27 includes a front optical system 29 shared with the first optical system 25, and a second rear optical system 31 located after the switching unit 32. The front optical system 29 includes, for example, at least one of a lens and a mirror, and forms an image of the object ob, which is the subject.

[0041] The switching unit 32 may be provided at or near the primary imaging position, which is the position at which the image of the object ob is formed by the front optical system 29 at a predetermined distance from the front optical system 29. The switching unit 32 has an action surface as onto which the electromagnetic wave that has passed through the front optical system 29 is incident. The action surface as is composed of a plurality of pixels px arranged two-dimensionally. The action surface as is a surface that causes an action, such as reflection or transmission, on the electromagnetic wave in at least one of a first state and a second state described below.

[0042] The switching unit 32 can switch, for each pixel px, between a first state in which the electromagnetic wave incident on the action surface as travels in a first direction d1 and a second state in which the electromagnetic wave travels in a second direction d2. The first state is a first reflection state in which the electromagnetic wave incident on the action surface as is reflected in the first direction d1. The second state is a second reflection state in which the electromagnetic wave incident on the action surface as is reflected in the second direction d2.

[0043] More specifically, the switching unit 32 includes a reflective surface that reflects electromagnetic waves for each pixel px. The switching unit 32 switches between a first reflective state and a second reflective state for each pixel px by changing the orientation of the reflective surface for each pixel px. In one embodiment, the switching unit 32 includes, for example, a DMD (Digital Micromirror Device). The DMD can drive tiny reflective surfaces that make up the acting surface as to switch the reflective surface for each pixel px to an inclined state at a predetermined angle, for example, +12° or −12°, with respect to the acting surface as. The acting surface as is parallel to the surface of a substrate on which the tiny reflective surfaces of the DMD are mounted.

[0044] The switching unit 32 switches between the first state and the second state for each pixel px under the control of the control unit 24. For example, the switching unit 32 can simultaneously switch some pixels px1 to the first state to cause the electromagnetic wave incident on the pixel px1 to travel in the first direction d1, and can simultaneously switch some other pixels px2 to the second state to cause the electromagnetic wave incident on the pixel px2 to travel in the second direction d2. Furthermore, by switching the same pixel px from the first state to the second state, the switching unit 32 can cause the electromagnetic wave incident on the pixel px to travel in the second direction d2 after the first direction d1.

[0045] 2, the first post-stage optical system 30 is disposed in a first direction d1 from the switching unit 32. The first post-stage optical system 30 includes, for example, at least one of a lens and a mirror. The first post-stage optical system 30 causes the electromagnetic wave, the traveling direction of which has been switched by the switching unit 32, to be incident on the detection element 26.

[0046] The second rear optical system 31 is disposed in the second direction d2 from the switching unit 32. The second rear optical system 31 includes, for example, at least one of a lens and a mirror. The second rear optical system 31 forms an image of the object ob as an electromagnetic wave whose traveling direction has been switched by the switching unit 32 on the detection surface of the image sensor 28.

[0047] With the above configuration, the detection device 21 can align the optical axis of the front optical system 29 with the optical axis of the first rear optical system 30 in the first direction d1 in which the electromagnetic wave travels in the first state, and with the optical axis of the second rear optical system 31 in the second direction d2 in which the electromagnetic wave travels in the second state. Therefore, the detection device 21 can reduce the parallax deviation between the depth detected by the detection element 26 in the case of an element array and the image captured by the imager 28 by switching the pixel px of the switching unit 32 between the first state and the second state.

[0048] As the reflecting unit 23 moves the irradiation position, the control unit 24 can switch some of the pixels px in the switching unit 32 to the first state and switch another part of the pixels px to the second state. Therefore, the detecting device 21 can cause the detecting element 26 to detect electromagnetic waves at some of the pixels px while simultaneously causing the imaging element 28 to detect images at another part of the pixels px. This allows the detecting device 21 to substantially simultaneously acquire images of the depth within the same field of view and of the part excluding the vicinity of the irradiation position of the electromagnetic waves.

[0049] 2, an optical system including a switching unit 32 is used to align or bring the detection optical axis of the distance measuring device 11 and the optical axis of the imaging device 12 close to each other. However, the method for aligning or bringing the optical axes of the distance measuring device 11 and the imaging device 12 close to each other is not limited to this. For example, the optical system 14 in FIG. 1 can be configured to substantially align the optical axis of the optical system of the distance measuring device 11 and the optical axis of the imaging device 12 using a dichroic mirror or a dichroic prism, taking advantage of the difference in wavelength between the light detected by the distance measuring device 11 and the image captured by the imaging device 12.

[0050] (Depth information processing device) The depth information processing device 13 according to an embodiment includes a distance information acquisition unit 41, an image acquisition unit 42, a control unit 43, a storage unit 44, and an output unit 45, as shown in FIG.

[0051] The distance information acquisition unit 41 acquires output data from the distance measuring device 11. The distance information acquisition unit 41 may include a communication module that communicates with the distance measuring device 11. In the case of the detection device 21 of FIG. 2, the distance information acquisition unit 41 acquires output data from the control unit 24 that constitutes part of the distance measuring device 11. The output data includes at least two-dimensional position information that indicates the detected position when the detection range is viewed in a plane, and a depth measurement value that is the distance to the object ob located at that position. The output data may further include information on the time when the depth was detected. The time information can also be added based on the time when the depth information processing device 13 acquired the output data.

[0052] The image acquisition unit 42 acquires images from the imaging device 12. The image acquisition unit 42 may include a communication module that communicates with the imaging device 12. In the case of the detection device 21 of FIG. 2, the image acquisition unit 42 acquires images from the control unit 24 that constitutes part of the imaging device 12. The images acquired by the image acquisition unit 42 may include information about the time when the image was captured. The time information may also be added based on the time when the depth information processing device 13 acquired the image. The time information of the output data of the distance measuring device 11 and the time information of the image need to be synchronized.

[0053] The control unit 43 includes one or more processors and memories, similar to the control unit 24 of the detection device 21. Similar to the control unit 24 of the detection device 21, the control unit 43 may include at least one of a general-purpose processor that loads a specific program and executes a specific function, and a dedicated processor specialized for a specific process.

[0054] The control unit 43 controls the entire depth information processing device 13, and also executes various processes for estimating a depth distribution at a predetermined timing based on the output data acquired by the distance information acquisition unit 41 and the image acquired by the image acquisition unit 42. The processes executed by the control unit 43 include estimating a depth distribution using machine learning, which will be described below.

[0055] The storage unit 44 includes a semiconductor memory and / or a magnetic memory. The storage unit 44 may function as, for example, a main storage unit, an auxiliary storage unit, or a cache memory. The storage unit 44 stores any information used in the operation of the depth information processing device 13. For example, the storage unit 44 may store a system program, an application program, a management database, etc. The storage unit 44 can temporarily store the output data acquired by the distance information acquisition unit 41, the images acquired by the image acquisition unit 42, and information obtained by processing these.

[0056] The storage unit 44 stores a trained model 46 generated by machine learning in another computer (hereinafter referred to as a "training computer") or the depth information processing device 13. The trained model 46 is generated by machine learning using a dataset that receives as input the output data acquired by the distance information acquisition unit 41 and preprocessed data of the image acquired by the image acquisition unit 42, and outputs a depth distribution at a predetermined timing. The trained model 46 may be stored in the form of a program and parameters. The control unit 43 can read out and use the trained model 46 from the storage unit 44.

[0057] The output unit 45 is configured to be able to output information to the outside from the depth information processing device 13. The output unit 45 includes one or more of a display, a communication module that transmits information to an external computer, a device that outputs information to a storage medium, etc.

[0058] (Depth distribution estimation process) The depth distribution estimation process executed by the control unit 43 will be described below with reference to the flowchart in Fig. 4. The depth information processing device 13 may be configured to read and implement a program recorded on a non-transitory computer-readable medium to execute the process executed by the control unit 43 described below. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media.

[0059] First, the control unit 43 acquires output data from the distance measuring device 11 via the distance information acquisition unit 41, and acquires an image captured by the imaging device 12 via the image acquisition unit 42 (step S101).

[0060] FIG. 5 is a diagram showing an example of a scene within the detection range of the depth distribution detection system 10. For example, the distance measuring device 11 and the imaging device 12 or the detection device 21 are fixed to a structure beside or above the road and detect a moving object MO traveling on the road. The moving object MO may be, for example, a car, a bicycle, or a pedestrian. Although FIG. 5 includes one moving object MO, the number of moving objects MO is not limited to one. In the detection range, the area other than the moving object MO becomes a background BG that does not change over time.

[0061] The frame rate and resolution of the ranging device 11 are lower than those of the imaging device 12. The period of one frame of the ranging device 11 is defined as a first period T1. For example, the frame rate of the ranging device 11 is 3 fps, and the first period T1 is 1 / 3 second. The number of depths that the ranging device 11 can measure during one frame can be, for example, 120 in the horizontal direction and 20 in the vertical direction, for a total of 2,400. For example, if the moving object MO is a car, the moving object MO moves within the field of view where the ranging device 11 performs measurement while the ranging device 11 scans one frame. Therefore, even if a depth distribution is generated by collecting depths obtained during the first period T1, it is not possible to obtain an instantaneous depth distribution at a specific time due to differences in measurement time. Therefore, the control unit 43 performs the following processing.

[0062] 6, the control unit 43 extracts multiple depths associated with time information for each of multiple second periods T2 obtained by dividing the first period T1 based on the output data of the distance measuring device 11 (step S102). The control unit 43 can, for example, divide the first period T1 into 100 second periods T2. For example, if the length of the first period T1 is 1 / 3 second, the length of the second period T2 can be 1 / 300 second. The number n by which the first period T1 is divided into the second periods T2 is not limited to 100 and can be any number.

[0063] As shown in FIG. 6, for each second period T2, the depth detected by the distance measuring device 11 during that second period T2 is extracted as depth information di. The depth information di is information that associates a depth measurement value with two-dimensional position information indicating the depth detection position and time information. A collection of multiple depth information di detected during each second period T2 is defined as a short-term depth information distribution SDk (k = 1 to n). For example, if the distance measuring device 11 measures a total of 2,400 depths, 120 in the horizontal direction and 20 in the vertical direction, during one frame, and n = 100, the number of depth information di included in each short-term depth information distribution SDk can be 24, 12 in the horizontal direction and 2 in the vertical direction. Since FIG. 6 is a diagram for explanatory purposes only, the number of depth information di in each short-term depth information distribution SDk is represented by a small number of dots.

[0064] 6, the distance measuring device 11 sequentially acquires depths at random positions within the detection range. However, the distance measuring device 11 may acquire depths by regularly scanning the detection range.

[0065] The control unit 43 may also generate information obtained by accumulating one frame of depth information di over the first period T1 as accumulated depth information AD. The accumulated depth information AD is information that, when the depth information di is associated with two-dimensional coordinates of the detection range and expressed as an image, results in an image that has distortions and shifts due to differences in measurement time.

[0066] The depth information di included in each short-term depth information distribution SDk (k = 1 to n) is assigned time information corresponding to the respective short-term depth information distribution SDk (k = 1 to n). As an example, the end time of the second period T2 may be assigned to the depth information di belonging to each short-term depth information distribution SDk (k = 1 to n). For example, if the start time of the first period T1 is set to 0, the time information t = T1 × k / n is assigned to the depth information di included in the short-term depth information distribution SDk (k = 1 to n). Note that if the output data output by the distance measuring device 11 includes information on the time at which the depth was measured, this information may be used as the time information of the depth information di. In this way, the measured depth value included in each depth information di is associated with a three-dimensional voxel including two-dimensional coordinates indicating the detection position when the detection range is viewed in a plane and time information.

[0067] Next, the control unit 43 generates area division information SIk (k=1 to n) (see FIG. 7) for dividing the detection range into an area of ​​the background BG and an area of ​​the moving object MO, based on the images acquired from the imaging device 12 for each of the second periods T2 (step S103). For this purpose, the control unit 43 extracts images at predetermined timings during the first period T1 from the images acquired from the imaging device 12. For example, as shown in FIG. 6, when the start time of the first period T1 is t=0, the control unit 43 may acquire an image IM1 at t=0 and an image IM2 at t=T1, which is the end time of the first period T1.

[0068] The control unit 43 estimates the position of the moving object MO at the end time of each second period T2 by using an optical flow method on the time-series images acquired from the imaging device 12. That is, as shown in FIG. 7, the control unit 43 extracts the moving object MO from the first image MI1 and extracts the same moving object MO (referred to as MO' in FIG. 7) from the second image MI2. The control unit 43 extracts feature points of the moving object MO and calculates a displacement vector v of the feature points between the first image MI1 and the second image MI2. The control unit 43 estimates the position of the moving object MO at the time corresponding to each second period T2, assuming that each feature point moves at a constant speed.

[0069] The control unit 43 determines region division information SIk (k=1 to n) based on the position of the moving object MO in each second period T2. The region division information SIk (k=1 to n) includes information on region boundary AB indicating the boundary between moving object region MA, which is the region where the moving object MO exists, and background region BA, which is the region of background BG other than moving object region MA. In FIG. 7, the region boundary AB is shown as a rectangle for the sake of explanation, but the region boundary follows the outline of the moving object MO when viewed in a plane.

[0070] The method of generating the region division information SIk (k = 1 to n) is not limited to using optical flow. For example, the control unit 43 can acquire an image captured at time t = T1·k / n corresponding to each second period T2 and calculate the difference between the image captured at time t = T1·k / n corresponding to each second period T2 and the image captured at time t = T1·k / n corresponding to each second period T2. The control unit 43 can determine that a region where the difference is substantially 0 is a background region BA because the image has not changed. The control unit 43 can also determine that a region where the difference is not substantially 0 is a moving object region MA because the image has changed. Note that a difference of substantially 0 means that the difference is 0 or a small value within the detection error range.

[0071] Furthermore, the control unit 43 can generate a luminance histogram for each pixel from multiple images, and generate region division information SIk (k=1 to n) based on this histogram. The histogram is expressed as a graph with luminance on the horizontal axis and frequency on the vertical axis. Based on multiple images, it can be determined that the position of a pixel where the frequency of a specific luminance is higher than a predetermined value corresponds to a background region BA. The position of a pixel where the luminance frequency is widely distributed can be determined to be a moving object region MA.

[0072] Furthermore, as shown in FIG. 8, the control unit 43 generates composite depth information CDk (k = 1 to n) for each second period T2, which includes accumulated depth information AD in the background region BA and multiple depth information di of the second period T2 in the moving object region MA (step S104).

[0073] That is, the control unit 43 applies the region division information SIk (k = 1 to n) generated in step S103 for each second period T2 to divide the detection range of the distance measuring device 11 into two-dimensional regions in a planar view. The control unit 43 generates composite depth information CDk (k = 1 to n) by using the depth information di of the short-term depth information distribution SDk (k = 1 to n) for the moving object region MA and the depth information di of the accumulated depth information AD for the background region BA. As a result, the background region BA of the composite depth information CDk (k = 1 to n) includes high-density depth information di that reflects the depth information di of one frame. In the background region BA, the detection target does not change over time, so a large number of depth information di accumulated over one frame can be used. On the other hand, in the moving object region MA, the depth information di changes over time, so low-resolution depth information di of each second period T2 is used. Note that, since FIG. 8 is a diagram for explanatory purposes only, the depth information di is represented by a small number of dots. The depth information di can be acquired at a higher density.

[0074] In this way, by increasing the density of the depth information di of the background area BA, the estimation accuracy of the depth distribution can be improved when the synthesized depth information CDk (k=1 to n) is used as an input for machine learning.

[0075] Next, the control unit 43 inputs the n pieces of composite depth information CDk (k = 1 to n) generated in step S104 into the trained model 46 to estimate a depth distribution at a predetermined timing (step S105). The predetermined timing can be set to any time during the first period T1. For example, the predetermined timing can be set to approximately the center of the first period T1. That is, when the start time of the first period T1 is t = 0, the predetermined timing can be set to approximately t = T1 / 2. For example, if the predetermined timing is equal to or greater than T1 × 0.4 and equal to or less than T1 × 0.6, the predetermined timing can be said to be approximately the center of the first period T1. The control unit 43 can store the estimated depth distribution at the predetermined timing in the memory unit 44. The control unit 43 can also output the estimated depth distribution at the predetermined timing from the output unit 45.

[0076] (Learning Computer) The trained model 46 used to estimate the depth distribution of the depth information processing device 13 is generated in advance by a training computer 50. The training computer 50 may be a computer different from the depth information processing device 13. Alternatively, the depth information processing device 13 may have the functions of the training computer 50. In one embodiment, the training computer 50 generates training data for machine learning through simulation, and generates the trained model 46 using this training data.

[0077] The training computer 50 moves a virtual moving object MO in a virtual three-dimensional space and calculates output data and images detected by a distance measuring device 11 and an imaging device 12 placed at predetermined positions in the virtual space. The training computer 50 can generate images using CG (Computer Graphics). The training computer 50 also calculates a depth distribution without delay at a predetermined timing by simulation calculation.

[0078] The learning computer 50 includes, for example, an input unit 51, a calculation unit 52, a storage unit 53, and an output unit 54, as shown in FIG.

[0079] The input unit 51 receives input of various data used in the simulation. The input unit 51 includes input devices such as a keyboard and a mouse, a communication module, and / or a storage medium reader. For example, the input unit 51 receives input of information such as the position and size of a stationary object or background placed in the virtual space. The input unit 51 also receives input of the position, size, movement direction, and speed of a moving object MO placed in the virtual space. Furthermore, the input unit 51 receives input of information on the position and orientation of the ranging device 11 and the imaging device 12 placed in the virtual space. Furthermore, the input unit 51 may receive settings of various conditions for performing the simulation. For example, the input unit 51 receives settings of a first period T1 and a second period T2.

[0080] The calculation unit 52 executes various calculations for generating the trained model 46. The calculation unit 52 includes one or more processors, similar to the control unit 24 of the detection device 21 and the control unit 43 of the depth information processing device 13. The calculation unit 52 may include a simulation unit 55 that performs simulation and a function approximator 56 that performs machine learning. The function approximator 56 estimates the relationship between input and output from training data, which is a data set combining multiple inputs and outputs. The function approximator 56 is, for example, a neural network. Note that in this embodiment, the simulation and machine learning are performed by the same training computer 50, but they may be executed by separate computers.

[0081] The storage unit 53 includes a semiconductor memory and / or a magnetic memory. The storage unit 53 can store any information used in the operation of the training computer 50 and any information output by the training computer 50. For example, the storage unit 53 can store various setting information for the simulation acquired from the input unit 51 and the trained model 46 generated as a result of machine learning.

[0082] The output unit 54 is configured to be able to output information to the outside from the learning computer 50. The output unit 54 includes a display, a communication module that transmits information to an external computer, a device that outputs information to a storage medium, and the like.

[0083] (Generating a trained model) The process of generating the trained model 46 executed by the calculation unit 52 will be described below using the flowchart in Figure 10. The training computer 50 may be configured to load a program recorded on a non-transitory computer-readable medium and implement all or part of the process executed by the calculation unit 52 described below. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media.

[0084] The calculation unit 52 generates, through simulation, the output data of the ranging device 11 and the image output by the imaging device 12 in the virtual space, as well as a depth distribution at a predetermined timing during the first period T1. Hereinafter, the depth distribution at the predetermined timing is referred to as the target depth distribution. The target depth distribution is a correct depth distribution at the predetermined timing, without including deviations due to differences in measurement times within one frame of the ranging device 11. The calculation unit 52 generates output data according to the frequency at which the ranging device 11 can actually measure depths. The calculation unit 52 may generate a target depth distribution including a greater number of pieces of depth information di than the number of pieces of depth information di that the ranging device 11 can actually detect during the first period T1, which is the duration of one frame. For example, if the ranging device 11 acquires depth information di at detection positions arranged 120 horizontally and 20 vertically during the first period T1, the target depth distribution calculated by the simulation may be a depth distribution at detection positions arranged 240 horizontally and 40 vertically. That is, the target depth distribution can include depth information di with a higher resolution than the depth information di that the distance measuring device 11 can actually detect.

[0085] Next, the calculation unit 52 extracts a plurality of pieces of depth information di associated with time information for each of the second periods T2 obtained by dividing the first period T1 from the output data of the distance measuring device 11 obtained as a result of the simulation (step S202). This process is the same as step S102 in the flowchart of Fig. 4. The first period T1 and second period T2 used here are set to be the same as the first period T1 and second period T2 used in step S102.

[0086] The calculation unit 52 generates area division information SIk (k=1 to n) for dividing the detection range into the area of ​​the background BG and the area of ​​the moving object MO for each second period T2 based on the image obtained as a result of the simulation (step S203). This process is the same as step S103 in the flowchart of FIG.

[0087] Furthermore, the calculation unit 52 generates, for each second period T2, synthesized depth information CDk (k=1 to n) that includes, in the background region BA, accumulated depth information AD obtained by accumulating the multiple pieces of depth information di during the first period T1, and includes, in the moving object region MA, multiple pieces of depth information di for the second period T2 (step S204). This process is the same as step S104 in the flowchart of FIG. 4.

[0088] That is, the calculation unit 52 performs the same processing on the output data of the distance measuring device 11 and the image of the imaging device 12 obtained by simulation as the processing performed by the depth information processing device 13 in the stage before performing inference by machine learning, to generate synthetic depth information CDk (k = 1 to n).

[0089] The calculation unit 52 generates training data including a plurality of combinations of composite depth information CDk (k=1 to n) corresponding to one frame of data and a target depth distribution (step S205). The calculation unit 52 may generate a plurality of data in parallel in the above steps S201 to S204. The calculation unit 52 generates a large number of data sets for machine learning, for example, including 1,000 to 10,000 sets of data.

[0090] The calculation unit 52 uses the function approximator 56 to perform machine learning using the composite depth information CDk (k = 1 to n) as input and the target depth distribution as output, thereby generating a trained model 46 (step S206). The calculation unit 52 stores the trained model 46 in the storage unit 53 and / or outputs it from the output unit 54. The trained model 46 can be used in multiple depth information processing devices 13.

[0091] (Machine learning calculation method) The function approximator 56 is, for example, a neural network.

number

number

number

number

[0092] The uncertainty represents the variance when the machine learning output is calculated as a probability expression of a normal distribution. The function approximator 56 performs a calculation to calculate the loss function L(θ) in Equation (2) and performs training to minimize this loss function L(θ).

[0093]

number

[0094] Here, i represents each data included in the training data. Di is the depth of the training data. The uncertainty used in equation (2) represents the degree of data difficulty in estimation.

[0095] The second term in Equation (2) acts to reduce uncertainty (variance). On the other hand, the exponential function exp() of the first term acts to increase uncertainty (variance). The calculation unit 52 performs learning so as to reduce the loss function L(θ), and therefore can perform learning by achieving a balance between the first and second terms. That is, according to this embodiment, uncertainty is taken into consideration when generating the trained model 46. By learning using uncertainty, it is expected that the output will be robust for depth measurements that are spatially far away and near object boundaries, temporally far from the target time, and for depth measurements that are difficult to estimate, such as for moving objects.

[0096] In this embodiment, as described above, the training computer 50 generates composite depth information CDk (k = 1 to n) as input for machine learning by adding time information with higher resolution than that of the first period T1 to the depth information di for each second period T2. The training computer 50 also generates a target depth distribution by simulation that does not include distortion due to a difference in measurement time, and outputs the target depth distribution as the machine learning output. By using the generated trained model 46, the depth information processing device 13 can input the composite depth information CDk (k = 1 to n) generated from the actual output data from the distance measuring device 11 and the image from the imaging device 12 into the trained model 46 to estimate a depth distribution at a predetermined timing. The estimated depth distribution reduces distortion and deviation in the depth distribution caused by a difference in measurement time by the distance measuring device 11.

[0097] Furthermore, in this embodiment, in both the learning stage by the learning computer 50 and the estimation stage by the depth information processing device 13, the background region BA includes accumulated depth information AD obtained by accumulating depth information di over the first period T1 as an input for machine learning. This allows for a higher density of input information than simply using short-term depth information distributions SDk (k = 1 to n) obtained by dividing one frame of ranging information into n pieces as input for machine learning. This improves the accuracy of machine learning, thereby improving the prediction accuracy of the depth distribution at a predetermined timing.

[0098] Furthermore, in this embodiment, in the learning stage of machine learning, calculations including uncertainty are performed to generate the learned model 46. This makes it possible to further reduce uncertainty even under position and time conditions where it is difficult to estimate the depth information di. This makes it possible to further improve the estimation accuracy of the depth distribution.

[0099] In fact, according to a simulation experiment conducted by the inventor of the present application, it was confirmed that the accuracy of the depth distribution estimation can be improved by dividing the depth information di, which is the input for machine learning, into time segments and adding time information. It was also confirmed that the accuracy of the depth distribution estimation can be improved by increasing the density of the background area BA by adding depth information di accumulated over one frame. Furthermore, it was confirmed that the accuracy of the depth distribution estimation can be improved by taking uncertainty into account during the learning stage of machine learning. It was also confirmed that the accuracy of the depth distribution estimation can be further improved by using these methods in combination.

[0100] Furthermore, according to this embodiment, the number of pieces of depth information di of the target depth distribution used as output in the learning stage is set to be greater than the number of pieces of depth information di output from the ranging device 11 as the depth information di of one frame in the first period T1. This allows the depth information processing device 13 to acquire, as the depth distribution at a predetermined timing, a depth distribution with higher density than the depth distribution that the ranging device 11 can acquire in one frame. This also allows the depth information processing device 13 to generate a depth map with higher resolution than the depth distribution output from the ranging device 11 when generating a depth map based on the depth distribution.

[0101] Although embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present disclosure. For example, functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined or divided into one. The embodiments of the present disclosure can also be realized as a method, a program executed by a processor included in an apparatus, or a storage medium on which a program is recorded. It should be understood that these are also included within the scope of the present disclosure. [Explanation of symbols]

[0102] 10 Depth distribution detection system 11 Ranging device 12 Imaging device 13 Depth information processing device 14 Optical system 21 Detection device 22 Irradiation unit 23 Reflector 24 Control Unit 25 First Optical System 26 Detector element 27 Second Optical System 28 Image sensor 29 Front optical system 30 First rear optical system 31 Second rear optical system 32 Switching section 41 Distance information acquisition section 42 Image acquisition unit 43 Control Unit 44 Memory section 45 trained models 50 Educational Computers 51 Input section 52 Arithmetic section 53 Storage section 54 Output section 55 Simulation Department 56 Function Approximators ax optical axis MO moving object BG background T1 First Period T2 Second Period IM1 First Image IM2 Second Image di depth information SDk(k=1~n) Short-term depth information distribution AD cumulative depth information SIk(k=1~n) Region division information CDk(k=1~n) Composite depth information

Claims

1. a distance information acquisition unit that acquires output data from a scanning distance measuring device; an image acquisition unit that acquires an image of a range that overlaps with the detection range of the distance measuring device; a processor that estimates a depth distribution at a predetermined timing within a first period, which is a period of one frame of the distance measuring device, based on the output data and the image, The processor: generating, from the output data, a plurality of pieces of depth information for each of a plurality of second periods obtained by temporally dividing the first period, the pieces of depth information including a depth, a position in a two-dimensional direction at which the depth was detected, and time information corresponding to the second period; generating area division information for dividing the detection range into a background area and a moving object area from the image for each of the second periods; generating, for each of the second periods, synthesized depth information including accumulated depth information obtained by accumulating the plurality of depth information during the first period in the background region and including the plurality of depth information during the second period in the moving object region; a processor that estimates the depth distribution by inputting the generated synthetic depth information into a trained model that has been trained using teacher data that receives a plurality of pieces of synthetic depth information for the second period and outputs a depth distribution at the predetermined timing within the first period; The trained model is generated by performing calculations that include uncertainty when learning using the training data, and the uncertainty represents the variance when the machine learning output is calculated as a probability representation of a normal distribution.

2. The depth information processing device according to claim 1 , wherein an optical axis of the optical system of the distance measuring device and an optical axis of the optical system that captures the image are substantially aligned.

3. The depth information processing device according to claim 1 or 2, wherein the region division information is generated from the images based on an optical flow, a difference between each of the images, or a luminance histogram obtained from a plurality of the images for the time series of the images.

4. The depth information processing device according to claim 1 , wherein the predetermined timing is a time substantially at the center of the first period.

5. The depth information processing device according to claim 1 , wherein the estimated depth distribution includes a number of pieces of depth information that is greater than a total number of pieces of depth information generated during the first period.

6. The depth information processing device according to claim 1 , wherein the synthesized depth information and the depth distribution of the training data are generated by executing a simulation in which a moving object is virtually placed in the detection range.

7. A depth distribution estimation method executed by a computer processor, comprising: Obtaining output data from the scanning range finder; Acquire an image capturing an area overlapping with the detection area of ​​the distance measuring device; generating, from the output data, a plurality of pieces of depth information for each of a plurality of second periods obtained by temporally dividing a first period, which is a period of one frame of the distance measuring device, the pieces of depth information including a depth, a position in a two-dimensional direction at which the depth was detected, and time information corresponding to the second period; generating area division information for dividing the detection range into a background area and a moving object area from the image for each of the second periods; generating, for each of the second periods, synthesized depth information including accumulated depth information obtained by accumulating the plurality of depth information during the first period in the background region and including the plurality of depth information during the second period in the moving object region; estimating the depth distribution by inputting the generated synthetic depth information into a trained model trained using training data that receives synthetic depth information for a plurality of the second periods and outputs a depth distribution at a predetermined timing within the first period; A method in which the trained model is generated by performing calculations including uncertainty when learning using the training data, and the uncertainty represents the variance when the machine learning output is calculated as a probability representation of a normal distribution.

8. a scanning distance measuring device; an imaging device that captures an image of a range that overlaps with the detection range of the distance measuring device; a depth information processing device including a processor that calculates a depth distribution at a predetermined timing within a first period, which is a period of one frame of the distance measuring device, based on output data of the distance measuring device and an image captured by the imaging device; and Equipped with The processor: generating, from the output data, a plurality of pieces of depth information for each of a plurality of second periods obtained by temporally dividing the first period, the pieces of depth information including a depth, a position in a two-dimensional direction at which the depth was detected, and time information corresponding to the second period; generating area division information for dividing the detection range into a background area and a moving object area from the image for each of the second periods; generating, for each of the second periods, synthesized depth information including accumulated depth information obtained by accumulating the plurality of depth information during the first period in the background region and including the plurality of depth information during the second period in the moving object region; estimating the depth distribution by inputting the generated synthetic depth information into a trained model trained using training data that receives a plurality of pieces of synthetic depth information for the second period and outputs a depth distribution at the predetermined timing within the first period; A depth distribution detection system in which the trained model is generated by performing calculations including uncertainty when learning using the training data, and the uncertainty represents the variance when the machine learning output is calculated as a probability representation of a normal distribution.

9. A computer-implemented trained model generation method, acquiring or generating a plurality of pieces of data combining a plurality of pieces of depth information and area division information generated for each of a plurality of second periods obtained by temporally dividing a first period, which is a period of one frame of a scanning type distance measuring device, and a depth distribution at a predetermined timing within the first period, wherein each piece of depth information includes a depth, a two-dimensional position at which the depth is detected, and time information corresponding to the second period, and the area division information divides a background area and a moving object area within a detection range of the distance measuring device; generating, for each of the second periods, synthesized depth information including accumulated depth information obtained by accumulating the plurality of depth information during the first period in the background region and including the plurality of depth information during the second period in the moving object region; generating a trained model by performing machine learning using training data in which the synthesized depth information for the plurality of second periods is input and the depth distribution is output; A trained model generation method in which, when generating the trained model using the training data, calculations including uncertainty are performed, and the uncertainty represents the variance when the output of machine learning is calculated as a probability representation of a normal distribution.

Images