Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

An information processing device has a parallax pattern acquisition unit and a scattering detection unit. The parallax pattern acquisition unit acquires a brightness distribution for a depth direction and a parallax direction that is orthogonal to the depth direction as assumed for a scattering environment as a parallax pattern. The scattering detection unit acquires a brightness distribution for the depth direction and the parallax direction as indicated by a RAW histogram from a LiDAR sensor as an actual measurement pattern. The scattering detection unit compares the actual measurement pattern with the parallax pattern to determine whether there is scattering within a sensing space.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Fusion systems are known that fuse data from LiDAR (Light Detection and Ranging) and stereo cameras to measure distance and recognize objects. Fusion systems are used in systems that require high-precision information, such as ADAS (Advanced Driver-Assistance Systems).

[0003] JP 2014-074708 A JP 2020-128925 A JP 2019-203774 A

[0004] Guy Satat, Matthew Tancik, Ramesh Raskar, "Towards Photography Through Realistic Fog", MIT Media Lab, [online], [Retrieved April 17, 2024], Internet <URL: https: / / web.media.mit.edu / ~guysatat / fog / materials / TowardsPhotographyThroughRealisticFog.pdf> T. Sang, Sung-You Tsai, TsungPo Y, "Mitigating Effects of Uniform Fog on SPAD Lidars", IEEE Sensors Letters, Volume: 4, Issue: 9, September 2020

[0005] To achieve safe autonomous driving, ADAS must be able to withstand bad weather. However, fusion systems can experience significant performance degradation in bad weather. This is because scattering by particles such as rain and fog degrades the output signals of upstream sensors (LiDAR and cameras), making it difficult to ensure the performance of downstream fusion systems.

[0006] To achieve high-precision sensing, it is necessary to detect the presence or absence of scattering within the sensing space and, if necessary, remove the scattering signal. A known scattering detection method is based on the LiDAR Z waveform (the shape of the RAW histogram in the depth direction). This method detects scattering based on the similarity between the Z waveform (theoretical waveform) expected in a scattering environment and the actual Z waveform (measured data). However, the waveform of the measured data fluctuates due to fluctuations in scattering density, so sufficient consistency with the theoretical waveform may not be achieved.

[0007] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that are capable of accurately detecting the presence or absence of scattering during bad weather.

[0008] According to the present disclosure, there is provided an information processing device including: a disparity pattern acquisition unit that acquires, as a disparity pattern, a luminance distribution in a depth direction and a disparity direction orthogonal to the depth direction assumed in a scattering environment; and a scattering detection unit that acquires, as an actual measurement pattern, a luminance distribution in the depth direction and the disparity direction indicated by a RAW histogram of a LiDAR sensor and compares the actual measurement pattern with the disparity pattern to determine the presence or absence of scattering in a sensing space. Also according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium that stores a program that causes a computer to realize the information processing of the information processing device.

[0009] 1 is a diagram illustrating an example of a system equipped with a fusion system. FIG. 1 is a diagram illustrating performance degradation during bad weather. FIG. 1 is a diagram illustrating performance degradation during bad weather. FIG. 1 is a diagram illustrating a performance improvement method for the fusion system of the present disclosure. FIG. 2 is a diagram illustrating an example configuration of an ADAS equipped with the fusion system of the present disclosure. FIG. 3 is a flowchart illustrating an overview of the processing of the fusion system. FIG. 4 is a diagram illustrating a specific flow from scattering detection to subject depth generation processing. FIG. 5 is a diagram illustrating a method for estimating a local scattering rate based on remaining power. FIG. 6 is a diagram illustrating a method for estimating a local scattering rate based on blurring of a received light image. FIG. 7 is a diagram illustrating a method for estimating a local scattering rate based on blurring of a received light image. FIG. 8 is a diagram illustrating an example of a method for extracting subject depth based on a local scattering rate. FIG. 9 is a diagram illustrating an example of data input to a LiDAR output unit. FIG. 10 is a diagram illustrating an example of data output from a LiDAR output unit. FIG. 11 is a diagram illustrating an example of luminance distribution of a scattering environment according to the parallax between a laser and a light receiving sensor. FIG. 12 is a diagram illustrating an example of a form of acquiring a parallax pattern. FIG. 13 is a diagram illustrating an example of a form of acquiring a parallax pattern. FIG. 14 is a diagram illustrating an example of internal estimation processing of a parallax pattern. FIG. 15 is a diagram illustrating an example of acquisition of a Z waveform feature value excluding a blind section. 1 is a diagram illustrating an example of scattering detection based on the results of matching a parallax pattern with an actual measurement pattern. FIG. 2 is a diagram illustrating correction of a scattering model. FIG. 3 is a diagram illustrating detection of scattering based on the results of projecting and receiving light from multiple lasers. FIG. 4 is a diagram illustrating an example of detection of scattering using multiple lasers. FIG. 5 is a diagram illustrating detection of scattering based on ambient light. FIG. 6 is a diagram illustrating the arrangement of a stationary light source. FIG. 7 is a diagram illustrating an example of an ambient light image. FIG. 8 is a diagram illustrating an example of a system configuration when ambient light is used. FIG. 9 is a diagram illustrating reduction of dead areas by expanding light emission channels. FIG. 10 is a diagram illustrating reduction of dead areas by expanding light emission channels. FIG. 11 is a diagram illustrating light intensity correction associated with light emission of multiple channels. FIG. 12 is a diagram illustrating light intensity correction associated with light emission of multiple channels. FIG. 13 is a diagram illustrating an example of a system configuration related to light intensity correction. FIG. 14 is a diagram illustrating reduction of dead areas by expanding light reception channels. FIG. 15 is a diagram illustrating reduction of dead areas by expanding light reception channels. FIG. 16 is a diagram illustrating improvement in estimation accuracy of a parallax pattern using history information. FIG. 17 is a diagram illustrating improvement in scattering detection accuracy by pre-extracting clear peaks.1A and 1B are diagrams illustrating an improvement in scattering detection accuracy by pre-extraction of clear peaks, and FIG. 1C is a diagram illustrating an example of the hardware configuration of a fusion system.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] The description will be given in the following order: [1. Overview] [1-1. Positioning of the fusion system] [1-2. Performance degradation in bad weather] [1-3. Performance improvement by estimating scattering rate distribution and restoring data] [2. Fusion system of the present disclosure] [2-1. Overview of processing] [2-2. Parallax pattern acquisition unit, feature acquisition unit, scattering detection unit] [2-3. Scattering rate estimation unit, scattering exclusion unit] [2-3-1. Estimation of local scattering rate based on remaining power] [2-3-2. Estimation of local scattering rate based on blurring of received light image] [2-3-3. Extraction of subject depth based on local scattering rate] [2-4. LiDAR output unit] [2-5. Fusion unit] [3. Specific form of scattering detection] [3-1. Luminance distribution of scattering environment according to parallax between laser and sensor] [3-2. Form of acquiring parallax pattern] [3-2-1. Internal estimation of disparity pattern] [3-2-2. Acquisition from ground truth pattern LUT] [3-3. Acquisition of Z waveform features excluding blind sections] [3-4. Detection of scattering based on the results of matching disparity pattern with actual measured pattern] [3-5. Correction of scattering model] [3-6. Detection of scattering based on the results of emitting and receiving light from multiple lasers] [3-7. Detection of scattering based on ambient light] [3-8. Reducing blind areas by expanding light emission channels] [3-9. Reducing blind areas by expanding light reception channels] [3-10. Improving estimation accuracy of disparity pattern using history information] [3-11. Improving scattering detection accuracy by pre-extracting clear peaks] [4. Example of hardware configuration] [5. Effects]

[0012] [1. Overview] [1-1. Positioning of the Fusion System] FIG. 1 is a diagram showing an example of a system equipped with a fusion system.

[0013] Fusion systems are used in systems that require highly accurate information, such as ADAS. ADAS has a two-stage configuration consisting of a sensor group and a fusion system. The sensor group, which is the first stage of the configuration, includes multiple ranging sensors such as cameras, LiDAR, and RADAR. Various sensor data detected by the sensor group is output to the fusion system, which is the second stage of the configuration. The fusion system uses sensor fusion to combine multiple sensor data and generate highly accurate output information.

[0014] In the example of Figure 1, the output information includes the corrected depth, the subject's movement, and the subject's recognition label. This output information is used for recognizing the surrounding environment, estimating the vehicle's position, and tracking objects. The vehicle control algorithm issues control instructions to various devices based on the output information.

[0015] [1-2. Performance Degradation in Bad Weather] FIGS. 2 and 3 are diagrams illustrating performance degradation in bad weather.

[0016] Optical sensors, such as cameras and LiDAR, that detect light and convert it into an electrical signal experience significant signal degradation at the time of output due to scattering and absorption from factors such as rain and fog. For example, in LiDAR measurements, the scattered light from scattering particles becomes stronger than the reflected light from the object, making the object undetectable. In camera measurements, the scattering of ambient light is superimposed in the depth direction, making distant objects appear hazy and less distinct. Combining degraded signals through sensor fusion does not produce accurate output information. While RADAR is highly resistant to bad weather, its low resolution means that sufficient performance cannot be guaranteed when relying primarily on RADAR.

[0017] For example, the scattering model of LiDAR can be expressed as the following equation (1): In equation (1), P(z) represents the power (residual power) at a distance z. ０represents the intensity of the laser light (laser emission intensity) at the time of laser irradiation. α(x, y, z) represents the local scattering rate at point (x, y, z). r represents a correction coefficient. (x, y, z) represents the coordinates of a point on a three-dimensional Cartesian coordinate system with the laser optical axis of the LiDAR sensor as the z axis.

[0018]

[0019] Hereinafter, the local scattering rate will be referred to as the "local scattering rate." The local scattering rate α is a parameter that indicates the concentration of scattering particles at each location. In this disclosure, the parameter that indicates the concentration of scattering particles is defined as the "scattering rate," but similar parameters include the "scattering coefficient" and the "transmittance." Any parameter that essentially indicates concentration is included in the concept of the "scattering rate."

[0020] In this disclosure, the scattering model shown in formula (1) is used, but a plurality of other scattering models are possible, such as those based on gamma distribution, etc. The method of this disclosure is not limited to the specific scattering model shown in formula (1), but can be applied to any scattering model that includes a concentration parameter.

[0021] [1-3. Performance Improvement by Estimating Scattering Rate Distribution and Restoring Data] FIG. 4 is a diagram illustrating a performance improvement technique for the fusion system of the present disclosure.

[0022] As described above, the residual power P(z) decreases rapidly as the distance z and the local scattering rate α increase. The signal from the object due to scattering weakens, while the signal from scattering particles strengthens, resulting in significant noise. In the present disclosure, this problem is ameliorated by estimating the scattering rate distribution and performing data reconstruction based on the estimation results.

[0023] For example, in equation (1), the remaining power P(z) is expressed by the product of three terms: "Item A" indicates the distance-squared attenuation term; "Item B" indicates the reflection ratio of the laser on the scattering particle; and "Item C" indicates the ratio of attenuation due to scattering when the light travels a distance z. For example, by applying the actual measurement results of the remaining power P(z) to equation (1), the scattering rate for each location (local scattering rate) can be obtained. By correcting the LiDAR data based on the spatial distribution of the scattering rate, LiDAR waveform information (subject depth) with reduced noise due to scattering can be obtained.

[0024] The scattering rate information can also be used to correct sensor data that is to be combined with LiDAR data LD by sensor fusion. In other words, the scattering rate information can also be used to correct sensor data prior to the actual sensor fusion. FIG. 4 shows an RGB image acquired by a camera as an example of sensor data to be combined. An RGB image refers to a color image having color information such as R (red), G (green), and B (blue). The RGB image is acquired, for example, as an image for distance measurement using a stereo imaging method.

[0025] For example, "item A" and "item B" in formula (1) are specific to LiDAR (active sensor) and do not apply to camera (passive sensor) data. On the other hand, "item C" indicates the attenuation rate according to the distance z and the local scattering rate α, and applies to both LiDAR and camera data. By correcting the RGB image based on the scattering rate information, it is possible to estimate an RGB image without noise due to scattering.

[0026] The scattering rate information can also be used to estimate the reliability of the sensor data. By adjusting the synthesis ratio in sensor fusion based on the reliability of each sensor data, the accuracy of the output information can be improved. In the following description, an RGB image from a camera is used as an example of the sensor data to be synthesized with the LiDAR data LD, but the sensor data is not limited to this. A similar method can also be applied when synthesizing sensor data from other optical sensors, such as an IR sensor or an EVS (Event-Based Vision Sensor).

[0027] 2. Fusion System of the Present Disclosure 2-1. Overview of Processing Fig. 5 is a diagram showing an example of the configuration of an ADAS 1 equipped with the fusion system 10 of the present disclosure. Fig. 6 is a flowchart showing an overview of processing by the fusion system 10.

[0028] The ADAS 1 includes a sensor group 30 and a fusion system 10. The sensor group 30 includes a LiDAR sensor 31 and a camera 32. For example, the LiDAR sensor 31 performs distance measurement using an image sensor with a single photon avalanche diode (SPAD) in each pixel. The camera 32 performs distance measurement using a stereo imaging method using multiple RGB images from different viewpoints. The sensor group 30 may also include other optical sensors, such as an IR sensor or EVS, instead of or in combination with the camera 32.

[0029] The fusion system 10 is an information processing device that combines various sensor data acquired from the sensor group 30 and generates highly accurate output information. For example, the fusion system 10 includes a scattering detection unit 11, a scattering rate estimation unit 12, a scattering exclusion unit 13, a LiDAR output unit 14, and a fusion unit 15.

[0030] The fusion system 10 performs sensor fusion processing according to the flow shown in Fig. 6. First, the fusion system 10 acquires various sensor data about the space to be sensed (sensing space) from the sensor group 30 (step S1). The sensor data includes LiDAR data LD (RAW data) output from the LiDAR sensor 31 and an RGB image IM (RAW data) output from the camera 32.

[0031] The scattering detection unit 11 analyzes the LiDAR data LD to determine whether scattering exists (step S2). If scattering does not exist (step S2: No), the fusion unit 15 synthesizes the LiDAR data LD and the RGB image IM by sensor fusion and generates a corrected depth DP. Ｒ (Step S6). The fusion system 10 generates the RGB image IM and the modified depth DP Ｒ The fusion system 10 generates data on the movement MO and the recognition label RL of the subject based on the various data generated (step S7).

[0032] If scattering is present (step S2: Yes), the scattering rate estimation unit 12 calculates the local scattering rate LS in the sensing space based on equation (1). Ｐ (Step S3) The local scattering rate LS Ｐ The scattering rate estimator 12 calculates the scattering rate at each point in the sensing space, thereby estimating the distribution of scattering rates within the sensing space from the LiDAR data LD. The scattering eliminator 13 eliminates components caused by scattering from the LiDAR data LD based on the scattering rate distribution, and extracts waveform information with the effects of scattering eliminated as the subject depth DP.

[0033] In order to reduce the amount of data, the LiDAR output unit 14 calculates the local scattering rate LS Ｐ The LiDAR output unit 14 compresses the data of the local scattering rate LS obtained by the compression. Ｂ The data is output to the fusion unit 15 together with the subject depth DP (step S4).

[0034] The fusion unit 15 calculates the local scattering coefficient LS Ｂ The fusion unit 15 estimates the amount of degradation of the RGB image IM based on the data (step S5). The fusion unit 15 corrects the RGB image IM based on the estimated amount of degradation, and generates an RGB image without noise due to scattering as a corrected image.

[0035] The fusion unit 15 calculates the reliability of each point of the corrected image based on the amount of degradation of the RGB image IM. The fusion unit 15 calculates the reliability of the subject depth DP of each point based on the signal strength of the subject depth DP and the amount of degradation of the RGB image IM. The fusion unit 15 determines a synthesis ratio for each point in sensor fusion based on the calculated reliability of each point. The fusion unit 15 performs sensor fusion based on the determined synthesis ratio, and converts the synthesized depth information into a corrected depth DP. Ｒ (step S6).

[0036] The fusion system 10 generates the corrected image and the modified depth DP. Ｒ The fusion system 10 generates data on the movement MO and the recognition label RL of the subject based on the various data generated (step S7).

[0037] The fusion system 10 determines whether there is data for the next frame (step S8), and if there is no data (step S8: No), the process ends. If there is data for the next frame (step S8: Yes), the process returns to step S1, and the fusion system 10 repeats the above process until there is no more data.

[0038] The following description focuses on the scattering detection and object depth DP generation processes of the information processing of the fusion system 10. Figure 7 shows a specific flow from scattering detection to object depth DP generation processes. The fusion system 10 has a parallax pattern acquisition unit 51 and a feature amount acquisition unit 52.

[0039] [2-2. Parallax Pattern Acquisition Unit, Feature Acquisition Unit, and Scattering Detection Unit] The scattering detection unit 11 determines the presence or absence of scattering based on a characteristic luminance distribution resulting from the parallax of the LiDAR sensor 31. "Parallax" refers to the deviation between the light projection position of the LiDAR light projector 35 (laser 35a) and the light reception position of the LiDAR light receiver 36 (light receiving sensor 36a). Because this deviation is similar to the difference in how an object appears at two observation points, in this disclosure, the deviation between the light projection position and the light reception position corresponding to the baseline length is referred to as "parallax," and the direction of the deviation between the light projection position and the light reception position is referred to as the "parallax direction." Hereinafter, the depth direction is referred to as the z direction, and the parallax direction is referred to as the x direction.

[0040] When LiDAR measurements are performed in a scattering environment, ideally, a z-direction brightness waveform (Z waveform) is obtained, in which brightness gradually decays due to scattering. Previously, it was proposed to detect scattering by modeling this Z waveform within a single pixel and extracting its features. However, in reality, a Z waveform is obtained in which brightness decays while repeatedly increasing and decreasing according to the concentration distribution of scattering particles (see the upper left diagram in Figure 14). The position of the increase and decrease fluctuates from moment to moment depending on changes in the concentration distribution. The larger the particle diameter, such as in rain or snow, the greater the concentration fluctuations. Therefore, conventional methods had low detection accuracy for scattering, posing a particular risk to distinguishing it from nearby subjects.

[0041] In this disclosure, to improve this low detection accuracy, a characteristic luminance distribution in the zx plane observed in a scattering environment (see the diagram at the bottom left of FIG. 14 ) is used as the analysis target. In a scattering environment, a light-dark pattern with an L-shaped boundary shape as shown in FIG. 14 occurs due to the parallax between the laser 35a and the light-receiving sensor 36a. The boundary shape of the pattern is determined by the hardware specifications of the LiDAR sensor 31 (the arrangement of the laser 35a relative to the light-receiving sensor 36a). The pattern occurs periodically in the parallax direction corresponding to each channel of the LiDAR sensor 31.

[0042] For example, the upper right diagram in Figure 19 shows the luminance distribution of fog. The lower right diagram in Figure 19 shows the luminance distribution of subject SB. While the signal from subject SB is localized, the signal from fog is widely distributed throughout the entire sensing space as long as it is not blocked by obstacles. As shown in Figure 14, due to parallax between the laser 35a and the light receiving sensor 36a, a blind area DA where sensing is impossible occurs near the surface of the LiDAR sensor 31. Within the zx plane, a bright pattern caused by scattering of fog and a dark pattern corresponding to the blind area DA occur, and the boundary between the bright pattern and the dark pattern is an L-shaped pattern boundary line BL (see Figure 17).

[0043] In a scattering environment, a luminance distribution in which light and dark are separated on either side of the pattern boundary line BL is obtained. The scattering detection unit 11 detects scattering based on whether the luminance distribution in the zx plane (measured pattern MP) indicated by the LiDAR data LD shows the luminance distribution specific to the scattering environment described above. Therefore, the presence or absence of scattering can be detected with higher accuracy than when a one-dimensional luminance distribution (Z waveform) in only the depth direction is used. When scattering is detected, the scattering detection unit 11 outputs a detection flag FL. In response to the detection flag FL, subsequent processing such as scattering rate estimation is performed.

[0044] For example, the parallax pattern acquisition unit 51 acquires, as a parallax pattern PT, a brightness distribution in the depth direction (z direction) and a parallax direction (x direction) orthogonal to the depth direction, which is assumed in a scattering environment. The parallax pattern PT refers to a light and dark pattern in the zx plane assumed according to the hardware specifications of the LiDAR sensor 31. The scattering detection unit 11 acquires, as a measured pattern MP, brightness distributions in the depth direction and parallax direction indicated by the RAW histogram of the LiDAR sensor 31. The scattering detection unit 11 compares the measured pattern MP with the parallax pattern PT to determine the presence or absence of scattering in the sensing space.

[0045] The hardware specifications may be acquired from a register of the LiDAR sensor 31 (register input) or may be estimated from the analysis results of the LiDAR data LD. The disparity pattern PT may be acquired from pre-registered information such as a look-up table (LUT), or may be acquired by internal calculation using a model function or the like (internal estimation). When performing internal estimation, the hardware specifications of the LiDAR sensor 31 may be estimated based on features (pattern features PF: see FIG. 17 ) extracted from the measured pattern MP, and the disparity pattern PT corresponding to the estimated hardware specifications may be obtained by internal calculation.

[0046] The feature amount acquisition unit 52 acquires the waveform in the depth direction (z direction) indicated by the RAW histogram as a Z waveform. The feature amount acquisition unit 52 acquires the feature amount of the Z waveform as a Z waveform feature amount WF. For example, the feature amount acquisition unit 52 acquires the depth direction position of the Z waveform peak (peak position), the peak brightness value (peak height), and the cumulative power P as the Z waveform feature amount WF. The scattering detection unit 11 scores the consistency between the Z waveform and the disparity pattern PT based on the Z waveform feature amount WF, and determines the presence or absence of scattering based on the calculated consistency score.

[0047] [2-3. Scattering Rate Estimation Unit / Scattering Removal Unit] The scattering rate estimation unit 12 estimates information about the scattering rate within the sensing space based on the sensing results of the LiDAR sensor 31. The LiDAR sensor 31 has a LiDAR light projecting unit 35 and a LiDAR light receiving unit 36. The LiDAR light projecting unit 35 emits laser light toward the sensing space. The LiDAR light receiving unit 36 ​​receives the laser light that has bounced off an object. The LiDAR light receiving unit 36 ​​converts the time it takes for the laser light to bounce back (time of flight) into a distance based on the speed of light.

[0048] The LiDAR light receiving unit 36 ​​outputs data on the brightness of the received laser light as LiDAR data LD. The LiDAR data LD is histogram data (RAW histogram) with the horizontal axis (z axis) representing distance and the vertical axis representing brightness L. The LiDAR light receiving unit 36 ​​sequentially outputs the LiDAR data LD to the feature amount acquisition unit 52 and the scattering rate estimation unit 12. When the disparity pattern PT is internally estimated, the LiDAR light receiving unit 36 ​​also sequentially outputs the LiDAR data LD to the disparity pattern acquisition unit 51.

[0049] The scattering rate estimation unit 12 sets a three-dimensional coordinate system in the sensing space with the distance direction as the z-axis, and acquires luminance data for each point in the sensing space from the LiDAR data LD. The scattering rate estimation unit 12 applies a scattering model to the three-dimensional luminance distribution and estimates the scattering rate distribution in the sensing space.

[0050] For example, the scattering rate estimation unit 12 includes a remaining power estimation unit 17, a scattering model correction unit 53, and a scattering rate and reliability calculation unit 18. The remaining power estimation unit 17 calculates the LiDAR data LD and the laser emission intensity P ０ The scattering model correction unit 53 corrects for variations in light intensity caused by parallax between the laser 35a and the light receiving sensor 36a. The scattering rate and reliability calculation unit 18 can estimate the distribution of scattering rates based on the corrected remaining power P. Hereinafter, the local scattering rate LS based on the remaining power P will be referred to as Ｐ The estimation method will be specifically explained below.

[0051] [2-3-1. Estimation of local scattering rate based on remaining power] FIG. 8 shows the local scattering rate LS based on the remaining power P. Ｐ FIG. 10 is a diagram illustrating a method for estimating the above.

[0052] The remaining power estimation unit 17 acquires the luminance data sequentially output from the LiDAR light receiving unit 36 ​​as LiDAR data LD. By sequentially acquiring the luminance data from the LiDAR light receiving unit 36, the luminance waveform gradually spreads in the z direction over time. Each time the remaining power estimation unit 17 acquires the luminance L data, it estimates the known laser emission intensity P ０From the waveform history of the luminance L and the luminance L, the remaining power P(z) at the distance z is successively estimated based on the following equation (2).

[0053]

[0054] In equation (2), K(L(Z)) is a conversion function that converts the luminance L into power (amount of light). This conversion function reflects, for example, the nonlinear SPAD sensitivity characteristic.

[0055] The scattering rate / reliability calculation unit 18 sequentially estimates the local scattering rate α(Z) from the luminance L(Z) and the remaining power P(Z) based on the following equations (3) and (4). The scattering rate / reliability calculation unit 18 acquires the scattering rate at each point in the sensing space as the local scattering rate α(Z). For each local scattering rate α(Z), the scattering rate / reliability calculation unit 18 calculates the degree of error that may be included in the calculation result of the local scattering rate α(Z) as the reliability.

[0056] For example, the scattering rate / reliability calculation unit 18 calculates the residual power P(Z) and the local scattering rate α(Z) by subtracting the rounding error ΔL of the luminance L. ｒｏｕｎｄ (Equation (5) and Equation (6) below), and the variation (standard deviation) Δα with the local scattering rate α (X±ΔX, Y±ΔY, Z±ΔZ) of the surrounding pixels. ＳＤ (Z), the reliability of the estimated local scattering rate α(Z) is calculated as Conf α (Z) is calculated (the following formula (7)).

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] In addition, in the formula (5), ΔP ｒｏｕｎｄ (Z) is the rounding error ΔL of the luminance L up to (Z-1). ｒｏｕｎｄ In equation (6), Δα ｒｏｕｎｄ (Z) is the rounding error ΔL of the luminance L ｒｏｕｎｄIn equation (7), g ａ (Δα ｒｏｕｎｄ (Z), Δα ＳＤ (Z)) denotes the score function of the reliability. ａ An example of the score function g is given in the following formula (8). ａ The shape of the score function g ａ may be set as in the following equation (9), and the reliability may be the estimation error itself.

[0063]

[0064]

[0065] [2-3-2. Estimation of local scattering rate based on blur of received light image] In the above example, the local scattering rate LS Ｐ was estimated based on the residual power P. However, the local scattering rate LS Ｐ can also be estimated based on the blur of the received light image. Ｐ FIG. 10 is a diagram illustrating a method for estimating the above.

[0066] When utilizing the blur of the received light image, it is necessary to use a different pixel driving method for the LiDAR light receiving unit 36 ​​than usual. For example, the scattering rate estimation unit 12 expands the driving region DR of the sensor surface (LiDAR light receiving unit 36) that is driven in synchronization with the laser to include surrounding pixels that the laser light does not normally reach. This allows a broad waveform (blurred waveform) resulting from the blur of the received light image to be detected across multiple pixels. The scattering rate estimation unit 12 derives the blurred waveform L of each point in the sensing space from the LiDAR data LD. ｂｏｋｅｈ Extract.

[0067] Blurred waveform L ｂｏｋｅｈincludes light reception data acquired from a pixel corresponding to the laser scanning position and one or more peripheral pixels that do not correspond to the laser scanning position. The peripheral pixels may be selected in any manner. For example, if the laser scanning position is a linear region with a width of one pixel, the peripheral pixels that are driven simultaneously with the laser scanning position may be a linear region with a width of one or more pixels outside the laser scanning position. The peripheral pixels that form the driving region DR do not necessarily have to be a linear region with a constant width. For example, the peripheral pixels that form the driving region DR may be arranged in a comb-like pattern outside the laser scanning position.

[0068] The scattering rate and reliability calculation unit 18 calculates the blurred waveform L ｂｏｋｅｈ The distribution of scattering rate is estimated based on the blurred waveform L ｂｏｋｅｈ (Z) can be expressed as the following formula (10) using a blurring function N(ΔX, α(Z)). The blurring function N(ΔX, α(Z)) is, for example, a normal distribution whose width is determined by the local scattering rate α(Z). The scattering rate / reliability calculation unit 18 estimates the local scattering rate α(Z) of each point in the sensing space by applying the actual measured value to formula (10).

[0069]

[0070] An advantage of this example is that the local scattering rate α(Z) can be estimated without using the waveform history of the luminance in the Z direction (residual power P). Note that in this example, the reliability of the local scattering rate α(Z) can also be calculated based on the luminance rounding error and the variation from surrounding pixels, as in the example of Fig. 8 .

[0071] 2-3-3. Extraction of Object Depth Based on Local Scattering Rate FIG. 11 is a diagram showing an example of a method for extracting an object depth DP based on the local scattering rate α(Z).

[0072] To improve the estimation accuracy of the local scattering rate α(Z), the scattering rate / reliability calculation unit 18 identifies signals from a spatial region where no objects (subject SB) other than scattering media such as rain or fog exist. The scattering exclusion unit 13 estimates the subject depth DP based on the identified signals. This is because if an object other than a scattering medium exists, reflected light from the object becomes dominant, making it difficult to accurately fit the luminance data to a scattering model.

[0073] For example, the scattering rate / reliability calculation unit 18 detects a section where signals from scattering media are estimated to be dominant as a scattering determination section SD based on the magnitude, rate of change, and reliability of the local scattering rate α(Z). The scattering exclusion unit 13 selectively uses the LiDAR data LD of sections other than the scattering determination section SD (non-scattering determination sections) to extract the subject depth DP.

[0074] In the example of FIG. 11 , when any of the following conditions is satisfied for the first time, the scattering rate / reliability calculation unit 18 detects the region in the sensing space up to the position in the z direction where the condition is satisfied for the first time as the scattering determination section SD. The scattering exclusion unit 13 determines the subject depth DP from the maximum peak of the section excluding the scattering determination section SD (non-scattering determination section). Condition 1: The value of the local scattering rate α(Z) is equal to or greater than a threshold. Condition 2: The rate of increase of the local scattering rate α(Z) is equal to or greater than a threshold. Condition 3: The reliability of the local scattering rate α(Z) is less than a threshold.

[0075] 12 and 13 are diagrams for explaining the processing of the LiDAR output unit 14. FIG. 12 shows data (local scattering rate LS) input to the LiDAR output unit 14. Ｐ 13 is a diagram showing an example of data (local scattering rate LS) output from the LiDAR output unit 14. Ｂ ) is a diagram showing an example.

[0076] The LiDAR output unit 14 outputs the local scattering rate LS Ｐ The feature quantity is extracted from the local scattering rate LS. ＢFor example, the LiDAR output unit 14 divides the sensing space into a plurality of blocks. The LiDAR output unit 14 outputs the local scattering rate LS of at least one block included in the scattering determination section SD among the plurality of divided blocks as Ｐ The representative value of the above is generated as correction information for correcting the RGB image IM.

[0077] The division may be performed in units of echo EC, or in units other than echo EC (for example, an arbitrarily set spatial unit). Echo EC refers to the peak waveform from its start to its end. If there are multiple peaks in a row, division can be performed at the valleys. In the example of FIG. 12, division is performed in units of echo EC. One echo EC constitutes one block. For example, the LiDAR output unit 14 divides the sensing space so that each of the multiple blocks includes at least one peak that appears in the waveform of the LiDAR data LD.

[0078] In the example of FIG. 12, N echoes EC are generated along the Z axis. "Echo1" to "Echo4" are echo groups indicating scattered echoes, and "Echo5" to "EchoN" are echo groups indicating subject echoes. Scattered echoes are echoes EC that exist in the scattering determination section SD and are primarily composed of scattered light from scattering media (rain, fog, etc.). Subject echoes are echoes EC that exist in sections other than the scattering determination section SD (non-scattering determination sections) and are primarily composed of reflected light from objects (subjects) other than scattering media.

[0079] The LiDAR output unit 14 outputs the local scattering rate LS Ｐ The LiDAR output unit 14 generates output data including the waveform data WV and the metadata MT and outputs the output data to the fusion unit 15.

[0080] The waveform data WV includes waveform information WI for each block. The waveform information WI refers to information about the features required to reproduce the waveform. For example, the waveform information WI includes information about the peak height of the waveform, the peak position of the waveform, the start position of the echo EC, the end position of the echo EC, and the half-width of the echo EC. The position refers to the position in the z-axis direction.

[0081] The metadata MT includes meta-information for each block. Meta-information refers to additional information about the waveform itself that accompanies the waveform in the block. For example, the meta-information includes a scattering determination label LB, scattering rate information SI, and scattering rate reliability information CF.

[0082] The scattering determination label LB is a label that indicates whether or not an echo EC in a block is a scattered echo. For example, a scattering determination label LB of "1" is assigned to a scattered echo. A scattering determination label LB of "0" is assigned to a subject echo. An echo EC with a scattering determination label LB of "0" is recognized as a scattered echo and treated as invalid data in the calculation of the subject depth DP. Note that the subject depth DP refers to the waveform information WI of an echo EC (subject echo) with a scattering determination label LB of "0" extracted from the waveform data.

[0083] The scattering rate information SI means a representative value of the local scattering rate in the echo section within the block (the section from the start position to the end position of the echo EC). The scattering rate reliability information CF means a representative value of the reliability of the local scattering rate in the echo section within the block. Examples of the representative value include the average value or the maximum value.

[0084] The fusion unit 15 calculates the local scattering coefficient LS Ｂ The data is used to perform correction and sensor fusion processing of the RGB image IM. In these processes, the scattering rate information SI and scattering rate reliability information CF of the scattered echo group, waveform information WI of the subject echo group (subject depth DP), and scattering determination label LB for identifying invalid information are essential.

[0085] Local scattering rate LS ＢOther information included in the waveform data WV and metadata MT can be used as verification information for sensor fusion for debugging and improving reliability. However, this information does not necessarily need to be output to the fusion unit 15. The LiDAR output unit 14 can also extract some of the information necessary for sensor fusion from the waveform data WV and metadata MT shown in FIG. 13 and selectively output only the extracted information to the fusion unit 15. Outputting only the necessary information reduces the amount of data transmitted.

[0086] For example, echoes EC in the non-scattering determination section are subject echoes resulting from light reflected from objects other than scattering media. Because the scattering rate is defined for the scattering medium, the scattering rate information SI of the subject echo has no physical meaning. Therefore, it is possible to output only the scattering rate information SI and scattering rate reliability information CF of echoes EC ("Echo1" to "Echo4") in the scattering determination section SD, and omit output of the scattering rate information SI and scattering rate reliability information CF of echoes EC ("Echo5" to "EchoN") in the non-scattering determination section as unnecessary information.

[0087] [2-5. Fusion Unit] The fusion unit 15 corrects the RGB image IM to be combined with the LiDAR data LD by sensor fusion based on the distribution of scattering rates before the sensor fusion. The fusion unit 15 performs sensor fusion on the RGB image IM (corrected image) corrected based on the distribution of scattering rates and the subject depth DP, and generates a corrected depth DP, which is high-precision depth information. Ｒ Generate.

[0088] 3. Specific Forms of Scattering Detection 3-1. Luminance Distribution of a Scattering Environment Depending on the Parallax Between the Laser and the Sensor FIG. 14 is a diagram showing an example of the luminance distribution of a scattering environment depending on the parallax between the laser 35a and the light receiving sensor 36a.

[0089] As described above, in a scattering environment, a light-dark pattern with an L-shaped boundary is observed due to the parallax between the laser 35a and the light-receiving sensor 36a. The shape of the pattern is determined by the hardware specifications related to the laser arrangement of the LiDAR sensor 31. The parallax pattern acquisition unit 51 acquires the light-dark pattern expected from the hardware specifications as the parallax pattern PT and determines the presence or absence of scattering by comparing the measured pattern MP with the parallax pattern PT.

[0090] The parallax pattern PT changes depending on the laser arrangement. The laser arrangement may include not only the positional relationship between the laser 35a and the light receiving sensor 36a (such as the amount of parallax), but also the number of lasers 35a and the irradiation control of the laser 35a (such as the irradiation range and irradiation timing). By performing scattering detection based on various laser arrangements, it is possible to improve detection accuracy.

[0091] 3-2. Acquisition of Parallax Pattern 3-2-1. Internal Estimation of Parallax Pattern FIGS. 15 and 16 are diagrams showing examples of acquisition of the parallax pattern PT.

[0092] 15 is an example of internally estimating the disparity pattern PT based on the LiDAR data LD. In the internal estimation process, first, hardware specifications related to the laser arrangement of the LiDAR sensor 31 are estimated based on the measured pattern MP. Then, the disparity pattern PT corresponding to the estimated hardware specifications is calculated based on a model function or the like.

[0093] FIG. 17 is a diagram showing an example of internal estimation processing of the disparity pattern PT based on the LiDAR data LD. The disparity pattern acquisition unit 51 acquires bins with valid brightness values ​​from the RAW histogram as effective brightness bins. "Valid" means that the bins are not at a noise level. The noise level criteria can be set arbitrarily by the system developer. The disparity pattern acquisition unit 51 acquires feature quantities related to the spatial arrangement of the effective brightness bins in the depth direction (z direction) and the parallax direction (x direction) as pattern feature quantities PF.

[0094] The feature quantity related to the spatial arrangement of the effective brightness bins means the geometric characteristics of the localized space of the effective brightness bins partitioned by the pattern boundary line BL. Therefore, the parallax pattern acquisition unit 51 can selectively acquire and use the effective brightness bins in the vicinity area (near view) of the LiDAR sensor 31 where the geometric characteristics of the pattern boundary line BL are likely to appear. The spatial range in which the effective brightness bins exist is determined by the laser arrangement. The laser arrangement can be determined by analyzing the shape of the localized space of the effective brightness bins.

[0095] For example, if the range of the foreground is Z≦Z_lim, the parallax pattern acquisition unit 51 extracts a pixel group P={(X, Y, Z)|L(X, Y, Z)≧L_TH} in the foreground that satisfies the luminance L(X, Y, Z)≧L_TH. The thresholds Z_lim and L_TH can be set arbitrarily by the system developer.

[0096] The pattern feature PF can be acquired as a feature of the pattern boundary line BL. For example, the disparity pattern acquisition unit 51 acquires, from the depth direction waveform (Z waveform) indicated by the RAW histogram, the bin closest to the foreground (-z side) where effective luminance values ​​begin to appear, as the effective luminance start bin. The disparity pattern acquisition unit 51 acquires, as the pattern boundary line BL, a line drawn by the effective luminance start bin in a plane including the depth direction (z direction) and the parallax direction (x direction). The disparity pattern acquisition unit 51 acquires, as the pattern feature PF, one or more singular points SP of the pattern boundary line BL.

[0097] Examples of the singular points SP include points (boundary points) located at the boundaries of each spatial local area LA and points (Z change points) where the direction of change in the z coordinate is reversed. A spatial local area LA refers to a detection area within the sensing space assigned to each channel of the LiDAR sensor 31. The sensing space is partitioned by a plurality of spatial local areas LA arranged in the parallax direction at an arrangement period according to the laser arrangement.

[0098] In a scattering environment, a light-dark pattern having an L-shaped boundary is observed for each spatial local area LA. Two singular points (an effective luminance start bin where the z coordinate is the maximum value and an effective luminance start bin where the z coordinate is the minimum value) that are intersections with the L-shaped pattern boundary line BL are obtained at both ends of each spatial local area LA in the x direction.

[0099] Based on the pattern feature PF, the disparity pattern acquisition unit 51 estimates hardware specifications related to the laser arrangement of the LiDAR sensor 31. The disparity pattern acquisition unit 51 acquires, as a disparity pattern PT, brightness distributions in the depth direction (z direction) and the disparity direction (x direction) estimated from the hardware specifications.

[0100] The laser arrangement to be estimated may include the number of lasers 35a, their positional relationship with the light receiving sensor 36a, and information related to the drive control of the lasers 35a. For example, the parallax pattern acquisition unit 51 acquires the arrangement period of the singular points SP in the parallax direction (x direction). Based on the arrangement period of the singular points SP and the position of each singular point SP, the parallax pattern acquisition unit 51 estimates the number of lasers 35a, the parallax amount and parallax direction of each laser 35a (whether the direction in which the laser 35a is arranged relative to the light receiving sensor 36a is the +x side or the −x side), the number of light-emitting channels per laser, and the light-receiving channels that are driven simultaneously with each light-emitting channel. The parallax pattern acquisition unit 51 acquires the estimated information as hardware specifications.

[0101] In addition, some information regarding the laser placement may be obtained from a register of the LiDAR sensor 31, or from light emission and light reception drive information that specifies drive conditions, etc.

[0102] Regarding the amount of parallax, for example, the boundary of the parallax pattern PT for each pixel can be calculated based on the geometric arrangement shown on the right side of Fig. 17, and the most consistent value can be adopted. In Fig. 17, "Θ_sen" indicates the pixel angle based on the sensor optical axis. "Θ_laser" indicates the laser angle based on the sensor optical axis. "SL_diff" indicates the amount of parallax (baseline length) between the laser 35a and the light receiving sensor 36a.

[0103] The z-coordinate of the bin (pattern boundary bin) on the pattern boundary line BL for each pixel is defined as "Edge(X, Y)." "Edge(X, Y)" can be expressed by the following formula (11): Edge(X, Y)=SL_diff×sin(90°−Θ_laser) / sin(Θ_sen+Θ_laser) (11)

[0104] The best matching value can be determined by the ratio of the number of valid luminance bins before and after the boundary, for example, as follows: The total number is obtained in XYZ space. - Ratio of valid luminance bins before the boundary: (Total number of valid luminance bins before Edge(X,Y)) / (Total number of bins before Edge(X,Y)) - Ratio of valid luminance bins after the boundary: (Total number of valid luminance bins after Edge(X,Y)) / (Total number of bins after Edge(X,Y))

[0105] The parallax pattern acquisition unit 51 adopts the parallax amount SL_diff that maximizes "Conf_pred" in the following formula (12): Conf_pred=1.0-(proportion of valid luminance bins before the boundary) / (proportion of valid luminance bins after the boundary) (12).

[0106] The disparity pattern acquisition unit 51 acquires the optimal value of "SL_diff" estimated by the above calculation as an estimated disparity amount. The disparity pattern acquisition unit 51 applies the estimated disparity amount to equation (11) to recalculate "Edge(X, Y)" and outputs it to a downstream processing block. The recalculated "Edge(X, Y)" indicates the position (z coordinate) of the pattern boundary line BL estimated from the LiDAR data LD.

[0107] The parallax pattern acquisition unit 51 acquires "Conf_pred" to which the estimated parallax amount is applied as the estimated reliability. The estimated reliability means the reliability of the pattern boundary line BL, which is scored based on the position of the pattern boundary line BL assumed in a scattering environment and the luminance distribution near the pattern boundary line BL. The parallax pattern acquisition unit 51 can output the estimated reliability to the scattering detection unit 11. The specified reliability can be used as one of the criteria for determining the consistency between the actual measurement pattern MP and the parallax pattern PT.

[0108] The above calculations may be performed as learning and inference processing by machine learning. For example, machine learning is performed using a raw histogram as input and a disparity pattern PT (the z-direction position of the pattern boundary line BL for each pixel) as output. Variations within the dataset may include variations in the type and concentration of scattering particles and hardware specifications related to laser placement (number of lasers, laser positions, number of emission channels, etc.). A convolutional neural network (CNN) may be used as the learning model.

[0109] [3-2-2. Acquisition from the Correct Pattern LUT] The example in FIG. 16 is an example of acquiring a disparity pattern PT using an LUT (lookup table). The disparity pattern acquisition unit 51 has a correct pattern LUT. The correct pattern LUT is a table that records the correct disparity pattern PT (correct pattern) for each hardware specification. The disparity pattern acquisition unit 51 acquires the hardware specification related to the laser arrangement from the LiDAR sensor 31 by register input. The disparity pattern acquisition unit 51 acquires the disparity pattern PT corresponding to the hardware specification of the LiDAR sensor 31 from the correct pattern LUT.

[0110] For example, a system developer pre-calculates the boundary of the disparity pattern PT for each pixel based on the geometric arrangement shown in the diagram on the right side of FIG. 17 from the hardware specifications at the time of LiDAR design (such as the arrangement and configuration of the LiDAR light-projecting unit 35 and the LiDAR light-receiving unit 36 ​​within the LiDAR housing, and the drive specifications of the LiDAR light-projecting unit 35 and the LiDAR light-receiving unit 36). Even during pre-calculation, the calculation method for "Edge (X, Y)" follows equation (11). The pre-calculated boundary of the correct disparity pattern PT is provided as an external input to the LiDAR light-receiving unit 36 ​​(SPAD) and used as a substitute for the above-mentioned disparity pattern PT.

[0111] 3-3. Acquisition of Z Waveform Feature Amount Excluding Dead Zone FIG. 18 is a diagram illustrating an example of acquisition of Z waveform feature amount WF excluding dead zone DZ.

[0112] The feature acquisition unit 52 acquires a dead zone DZ where sensing is not possible based on the parallax pattern PT, and can exclude the raw histogram data of the dead zone DZ from the acquisition target of the Z waveform feature value WF. The dead zone DZ refers to a data zone in the depth direction (z direction) that indicates the dead area DA. The data of the dead zone DZ is noise, and the Z waveform feature value WF cannot be acquired. The feature acquisition unit 52 analyzes the Z waveform not from the origin (z = 0), but from a position that is retreated from the origin by the dead zone DZ.

[0113] The feature acquisition unit 52 sets a section of a predetermined length after the dead zone DZ as a feature acquisition section. The feature acquisition section refers to a section from which the Z waveform feature WF is acquired. The length of the feature acquisition section can be set arbitrarily by the system developer based on empirical rules, a scattering model, or the like.

[0114] For example, the length in the z direction of the feature acquisition section is "Z_len", and the z coordinate of the bin (pattern boundary bin) on the pattern boundary line BL at pixel (X, Y) is "Edge(X, Y)". In this case, the feature acquisition section is the range shown in the following formula (13). The feature acquisition unit 52 acquires features such as peak height, peak position (e.g., max value, argmax value), and cumulative power (sum of luminance) in the feature acquisition section as Z waveform feature WF. Edge(X, Y)≦Z≦Edge(X, Y)+Z_len (13)

[0115] 3-4. Detection of Scattering Based on the Matching Result of Parallax Pattern and Measured Pattern FIG. 19 is a diagram showing an example of detection of scattering based on the matching result of the parallax pattern PT and the measured pattern MP.

[0116] The scattering detection unit 11 compares the parallax pattern PT with the RAW histogram and evaluates the consistency between them. Features that are evaluation factors tend to appear near the pattern boundary line BL. Therefore, the scattering detection unit 11 can determine scattering by scoring the consistency between the Z waveform near the pattern boundary line BL and the parallax pattern PT.

[0117] For example, the scattering detection unit 11 combines the ratio of pixels (X, Y) where an effective brightness bin exists near the pattern boundary line BL, the difference in the z direction between the pattern boundary line BL and the peak position "Zpeak", and a comparison of the peak height "L" and the cumulative power "Zpwr", and calculates a score for consistency with the characteristics expected in the case of scattering. The scattering detection unit 11 determines the presence or absence of scattering based on the calculated consistency score.

[0118] For example, the consistency score can be expressed based on the following evaluation elements, "Element 1" to "Element 3." "Score 1" to "Score 3" indicate the scores for "Element 1" to "Element 3." "Total number of XY pixels" indicates the total number of pixels of the light-receiving sensor 36a.

[0119] <Element 1: Proportion of pixels in which a valid intensity bin exists near the pattern boundary line> Score1 = N_edge / Total number of pixels in X and Y (14)

[0120] In equation (14), "N_edge" is the total number of pixels having a valid brightness bin in the vicinity of the pattern boundary line BL. The range in the vicinity of the pattern boundary line BL can be expressed by the following equation (15). The range of valid brightness L can be expressed by the following equation (16). Note that the thresholds "Z_error" and "L_TH" can be set arbitrarily by the system developer. Edge(X,Y)≦Z≦Edge(X,Y)+Z_error (15) L(X,Y,Z)≧L_TH (16)

[0121] <Element 2: Difference in z direction between pattern boundary line and peak position> Score2 = Σ Ｘ，Ｙ {1.0 / |Zpeak(X,Y)-Edge(X,Y)|} / total number of pixels in X and Y (17)

[0122] <Element 3: Comparison of peak height and cumulative power> Score 3 = Σ Ｘ，Ｙ {F1(L(X,Y,Zpeak),Zpwr)} (18)

[0123] In equation (18), "F1(L, Zpwr)" is an arbitrary function that evaluates the cumulative power "Zpwr" based on the peak height "L". For example, "F1(L, Zpwr)" is expressed by the following equation (19), but "F1(L, Zpwr)" may be arbitrarily set by an external input. F1(L, Zpwr)=Zpwr / L (19)

[0124] <Consistency Score> The consistency score "DetScore" can be calculated using "Score1" to "Score3" as shown in the following formula (20). In formula (20), "C1" to "C3" are coefficients indicating weights. "C1" to "C3" can be set arbitrarily by the system developer. DetScore = C1 x Score1 + C2 x Score2 + C3 x Score3 (20)

[0125] The scattering detection unit 11 outputs a detection flag FL when "DetScore" satisfies the condition of the following formula (21). The threshold value "DetThresh" can be set arbitrarily by the system developer. DetScore≧DetThresh (21)

[0126] In calculating the consistency score "DetScore," the estimation reliability at the time of internal estimation of the disparity pattern PT may be included in the consistency evaluation. The estimation reliability means the value of "Conf_pred" when the disparity amount SL_diff (the most accurate disparity amount estimated from the RAW histogram) that maximizes "Conf_pred" in equation (12) is adopted.

[0127] The estimation reliability means the reliability of the pattern boundary line BL, which is scored based on the position of the pattern boundary line BL assumed in a scattering environment and the luminance distribution in the vicinity of the pattern boundary line BL. For example, the scattering detection unit 11 acquires the estimation reliability from the parallax pattern acquisition unit 51. The scattering detection unit 11 determines the presence or absence of scattering based on the consistency score "DetScore" that takes into account the score "Score4" related to the estimation reliability. "Score4" can be expressed by the following formula (22). Score4=Conf_pred (22)

[0128] In calculating the consistency score "DetScore," the consistency evaluation may include a reliability based on the difference between the disparity pattern PT (correct pattern: Edge_GT(X,Y)) pre-registered in the correct pattern LUT and the disparity pattern PT (estimated pattern: Edge(X,Y)) obtained by internal estimation.

[0129] For example, the scattering detection unit 11 acquires a correct disparity pattern PT that is pre-registered in a correct pattern LUT based on the hardware specifications of the LiDAR sensor. The scattering detection unit 11 determines the presence or absence of scattering based on a consistency score "DetScore" that takes into account a reliability score "Score5" based on the difference between the disparity pattern PT (estimated pattern) estimated from the RAW histogram and the pre-registered disparity pattern PT (correct pattern). "Score5" can be expressed by the following formula (23): Score5=Σ Ｘ，Ｙ {1.0 / |Edge_GT(X,Y)-Edge(X,Y)|} (23)

[0130] For example, the scattering detection unit 11 can determine the presence or absence of scattering based on the consistency score "DetScore" corrected by the correction value "DetScore+" expressed by the following formula (24). In formula (24), "C4" and "C5" are coefficients indicating weights. "C4" and "C5" can be set arbitrarily by the system developer. DetScore+=C4×Score4+C5×Score5 (24)

[0131] In the example of Figure 19, the fog signal and the signal of the subject SB are compared with the disparity pattern PT (estimated pattern) by internal estimation. The fog signal shows high similarity to the estimated pattern. Both "Score 4" and "Score 5" are high, and as a result, the consistency score "DetScore" is also high. The signal of the subject SB is significantly different from the estimated pattern. Both "Score 4" and "Score 5" are low, and as a result, the consistency score "DetScore" is also low.

[0132] The above calculations may be performed as learning and inference processing by machine learning. For example, machine learning is performed using a raw histogram as input and a disparity pattern PT (the position of the pattern boundary line BL for each pixel in the z direction) and a detection flag FL as output. Variations within the dataset may include variations in the type and concentration of scattering particles and hardware specifications related to laser placement (number of lasers, laser position, number of emission channels, etc.). A CNN may be used as the learning model.

[0133] 3-5. Correction of Scattering Model FIG. 20 is a diagram for explaining the correction of the scattering model.

[0134] The scattering model correction unit 53 corrects the estimated scattering model to eliminate the influence of light intensity fluctuations caused by generation of the parallax pattern PT and the influence of the dead zone DZ. The correction target is the function f shown in equation (4). For example, the remaining power estimation unit 17 estimates the scattering model (function f) according to the Z waveform feature value WF. The scattering model correction unit 53 calculates the gain of the scattering model based on the parallax pattern PT, and acquires the scattering model corrected by the gain as a corrected scattering model (function f').

[0135] Simply put, by reusing the estimated model for a pixel where the influence of parallax is minimum for surrounding pixels (within the spatial local region LA), it is possible to obtain a corrected scattering model (function f') with reduced influence of parallax. For example, the scattering model correction unit 53 determines the pixel (X', Y') that becomes min{Edge(X, Y)} for each spatial local region "region_i" (i = 1, 2, 3, ...). The scattering model correction unit 53 reuses the model of pixel (X', Y') for the surrounding pixels in accordance with the following equations (25) and (26), thereby avoiding the influence of parallax as much as possible.

[0136] f'(X, Y, Z) = Invalid (Z < Edge(X, Y)) (25) f'(X, Y, Z) = f(X', Y', Z) (Z ≥ Edge(X, Y). However, both (X, Y) and (X', Y') belong to the same spatial local region "region_i") (26)

[0137] "Invalid" in equation (25) represents an invalid tag. In the range of Z<Edge(X,Y), L(Z)=0 and P(X,Y,Z)=0, and f(X,Y,Z,α) and the associated α(X,Y,Z) cannot be defined. Therefore, the invalid tag "Invalid" is assigned as the value of the function f'.

[0138] As shown in Fig. 20, a light intensity correction coefficient may be estimated according to the parallax pattern PT, and a corrected scattering model (function f') may be obtained after performing gain correction. For example, as shown in the following equations (27) and (28), laser light intensity correction may be performed according to the scattering model (squared attenuation x exponential attenuation) to correspond to the signal loss in front of the pattern boundary line BL. Regarding gain, a light intensity correction coefficient may be given as an external input instead of calculation using the function "Gain". P'(X,Y,Z) = P(X,Y,Z) x Gain(Edge(X,Y)) (27) Gain(Edge(X,Y)) = Edge(X,Y) －２ ×exp(-α(X,Y,Z)×Z) (28)

[0139] The scattering model correction unit 53 obtains a corrected scattering model (function f') as shown in the following equations (29) to (31) using the remaining power P' after correction in equation (27): f'(Z) = Invalid (Z < Edge(X, Y)) (29) L(Z) = P'(X, Y, Z) × (f(Z, α') / f(Z-1, α')) (Z ≥ Edge(X, Y)) (30) f'(Z) = r × α'(X, Y, Z) / Z ２ ×exp(-α′(X,Y,Z)×Z) (Z≧Edge(X,Y)) (31)

[0140] 3-6. Detection of Scattering Based on the Results of Projecting and Receiving Light from Multiple Lasers FIG. 21 is a diagram for explaining detection of scattering based on the results of projecting and receiving light from multiple lasers 35a.

[0141] The LiDAR light projector 35 may include multiple lasers 35a. Using multiple lasers 35a allows for the generation of various brightness distributions. The brightness distribution becomes more complex by varying the parallax amount, power, drive timing, etc. of each laser 35a. The settings for the parallax amount, drive timing, etc. are reflected in "Edge(X, Y)" in equation (11). The more complex the pattern, the more accurate the scattering detection.

[0142] For example, the parallax pattern acquisition unit 51 acquires a parallax pattern PT for each laser 35 a based on the hardware specifications of the LiDAR sensor 31 including the plurality of lasers 35 a. The scattering detection unit 11 compares the plurality of parallax patterns PT corresponding to the plurality of lasers 35 a with the actual measurement pattern MP to determine the presence or absence of scattering.

[0143] The LiDAR light projecting unit 35 can include a high-power laser and a low-power laser as the multiple lasers 35a. Measurements using only high-power lasers can accurately measure distant objects, but the signal from nearby scattered light is likely to saturate. Measurements using only low-power lasers can accurately measure nearby scattered light signals, but it is difficult to accurately measure distant objects. By using both high-power and low-power lasers, the shortcomings of each measurement method can be compensated for, allowing for accurate measurement over a wide area.

[0144] Note that "high power" and "low power" are relative terms and do not define numerical ranges. In this example, of two lasers with different laser powers, the laser with the relatively higher power is referred to as the "high power laser," and the laser with the relatively lower power is referred to as the "low power laser."

[0145] 22 and 23 are diagrams showing an example of detecting scattering using a plurality of lasers 35a.

[0146] The multiple lasers 35a emit laser light at different times. The individual RAW histograms derived from the lasers 35a are output sequentially to the scattering detection unit 11 without being combined. The scattering detection unit 11 acquires a parallax pattern PT for each laser 35a according to the laser arrangement. The parallax pattern PT can be acquired, for example, from a ground truth pattern LUT.

[0147] The scattering detection unit 11 acquires a plurality of measured patterns MP obtained by shifting the irradiation timing of the plurality of lasers 35a. The scattering detection unit 11 compares the acquired plurality of measured patterns MP with the corresponding parallax patterns PT. The scattering detection unit 11 combines the individual comparison results obtained for each parallax pattern PT to determine whether scattering is present or not.

[0148] "Individual matching results are integrated to determine whether scattering exists or not" means that the consistency scores from each laser 35a obtained by matching are evaluated independently or in combination, and the individual evaluation results are combined with "AND" or "OR" to form the basis for determining scattering.

[0149] In the example of Fig. 23, the scattering detection unit 11 calculates a consistency score "DetScore1" based on the irradiation result of "Laser 1" alone. The scattering detection unit 11 calculates a consistency score "DetScore2" based on the irradiation result of "Laser 2" alone. The scattering detection unit 11 evaluates "DetScore1" and "DetScore2" independently based on the threshold value "DetThresh".

[0150] The scattering detection unit 11 assigns weights to "DetScore1" and "DetScore2" and combines them. The scattering detection unit 11 evaluates the combined score obtained by the combination based on a threshold value "DetThresh_joint." "D1" and "D2" shown in FIG. 23 are coefficients indicating weights.

[0151] The scattering detection unit 11 combines the evaluation results of "DetScore1" alone, the evaluation results of "DetScore2" alone, and the evaluation results of the integrated score using logical expressions such as "AND" and "OR." The scattering detection unit 11 determines the presence or absence of scattering based on the evaluation results obtained by the combination. When combining the evaluation results, it is also possible to weight each individual evaluation result.

[0152] The fusion system 10 of this example includes a histogram adder 54 before the remaining power estimator 17. The histogram adder 54 adds the individual RAW histograms derived from each laser 35a to generate a composite histogram. The remaining power estimator 17 obtains information about the remaining power P based on the composite histogram.

[0153] 3-7. Detection of Scattering Due to Ambient Light FIG. 24 is a diagram illustrating detection of scattering due to ambient light EL.

[0154] Light sources other than the laser 35a can also be used for scattering detection as long as they cause asymmetry in the shape of the dead zone DA based on the light source placement. A stationary light source 39 that continuously emits light at a constant brightness is preferable because it does not produce a brightness distribution that becomes noise. The stationary light source 39 simply outputs light to which the light receiving sensor 36a is sensitive (e.g., halogen light including the 905 nm band) as ambient light EL. The stationary light source 39 does not need to be constantly on as long as its brightness does not fluctuate during measurement.

[0155] FIG. 25 is a diagram showing the arrangement of the stationary light source 39. As shown in FIG.

[0156] Because the shape of the dead zone DA needs to be acquired in advance, it is desirable that the positional relationship between the stationary light source 39 and the LiDAR sensor 31 be fixed. To clarify the shape of the dead zone DA, a light source with a certain degree of directionality is used for the stationary light source 39. For example, the position of an in-vehicle light source such as a headlight is fixed relative to the in-vehicle LiDAR sensor 31. Therefore, the in-vehicle light source can be used as the stationary light source 39 for scattering detection.

[0157] FIG. 26 is a diagram showing an example of the ambient light image EI.

[0158] The ambient light image EI is an image showing the luminance distribution caused by ambient light EL. The luminance of the ambient light EL is acquired as a stationary component of a RAW histogram. When a stationary light source 39 is installed at the center point (X0, Y0), a bright pattern is generated in the ambient light image EI that is concentrated in a specific direction from the center point (X0, Y0) based on the direction and strength of the directivity of the stationary light source 39. If the light and dark pattern caused by the ambient light EL can be acquired based on pre-registered information (correct pattern LUT) or internal estimation, the presence or absence of scattering can be determined by comparing it with the ambient light image EI obtained by actual measurement.

[0159] FIG. 27 is a diagram showing an example of a system configuration when ambient light EL is used.

[0160] The fusion system 10 of this example has an ambient light acquisition unit 55 located before the scattering detection unit 11. The ambient light acquisition unit 55 acquires the luminance distribution caused by ambient light indicated by the RAW histogram as an ambient light image EI. The scattering detection unit 11 compares the ambient light image EI with the luminance distribution caused by ambient light expected in a scattering environment to determine the presence or absence of scattering.

[0161] Scattering detection using the fixed light source 39 can be performed alone, or can be used in combination with scattering detection using the laser 35a. For example, the scattering detection unit 11 calculates the consistency score derived from ambient light in the same manner as when the laser 35a is used. The scattering detection unit 11 evaluates the consistency score derived from ambient light and the consistency score derived from laser light independently or in combination, and can combine the evaluation results with "AND" or "OR" to use them as the basis for scattering determination.

[0162] The specific procedure is as follows: First, the scattering detection unit 11 evaluates the luminance at the center point (X0, Y0) based on the following formula (32). In formula (32), "AmbiLw" and "AmbiUp" indicate luminance thresholds for determining whether or not the fixed light source 39 is present. AmbiLw≦Ambi(X0, Y0)≦AmbiUp (32)

[0163] When the luminance at the center point (X0, Y0) is between the lower limit "AmbiLw" and the upper limit "AmbiUp", the scattering detection unit 11 scores the ambient light image EI based on the following equations (33) to (35): AmbiScore1=Σ ｒ |ΔAmbi(X,Y)| (33) AmbiScore2=(Σ Ｘ |ΔAmbi(X,Y0)|+Σ Ｙ |ΔAmbi(X0,Y)|) / 2 (34) DetScore_ambi=1.0-AmbiScore1 / AmbiScore2 (35)

[0164] In equation (33), "r" indicates the radial direction (direction of directivity) in the xy plane where the bright patterns are concentrated. "ΔAmbi" indicates the difference in ambient light per pixel. The scattering detection unit 11 obtains the average value of the brightness difference in the radial direction using equation (33). The scattering detection unit 11 obtains the average value of the brightness difference in each of the x and y directions as a reference value using equation (34). "DetScore_ambi" indicates the consistency score for the ambient light image EI. The smaller the gradient in the radial direction, the higher the score.

[0165] The scattering detection unit 11 calculates the consistency score "DetScore_laser" derived from the laser light by the above-mentioned method, and evaluates "DetScore_laser" based on the following formula (36). In formula (36), "DetThresh_laser" is a threshold value that serves as an evaluation criterion. DetScore_laser≧DetThresh_laser (36)

[0166] The scattering detection unit 11 evaluates "DetScore_ambi" based on the following formula (37). In formula (37), "DetThresh_ambi" is a threshold value that serves as an evaluation criterion. DetScore_ambi≧DetThresh_ambi (37)

[0167] The scattering detection unit 11 combines "DetScore_laser" and "DetScore_ambi" with weights based on the following formula (38). The scattering detection unit 11 evaluates the combined score obtained by the combination based on a threshold value "DetThresh_joint". In formula (38), "D1" and "D2" are coefficients indicating weights. D1 × DetScore_laser + D2 × DetScore_ambi ≧ DetThresh_joint (38)

[0168] 3-8. Reduction of Dead Areas by Expanding Light Emission Channels FIGS. 28 and 29 are diagrams for explaining reduction of dead areas DA by expanding light emission channels.

[0169] In LiDAR measurement, multiple channels are often provided to divide the laser emission area. In such cases, it is common for the light receiving sensor 36a to also control pixel drive on a channel-by-channel basis in conjunction with the light receiving sensor 36a. Normally, light emission and light reception are controlled for each channel. However, in this example, instead of driving light-emitting channels and light-receiving channels in a one-to-one correspondence, multiple light-emitting channels are simultaneously driven for one light-receiving channel.

[0170] For example, a pair of light-emitting and light-receiving channels linked as the same channel is referred to as the "own channel." Normally, only the light-emitting channel of the own channel is driven. However, in this example, the light-emitting channel of another channel that faces closer to the foreground than the own channel is also driven simultaneously. By simultaneously driving multiple light-emitting channels, the blind area DA is reduced and detection accuracy is improved (see FIG. 29).

[0171] The number of simultaneously driven separate channels may be one or more. In the example of Figure 28, two light-emitting channels whose laser irradiation areas are adjacent to each other are driven simultaneously. As the number of simultaneously driven light-emitting channels increases, the dead area DA is further reduced. It is preferable that the laser irradiation areas of each simultaneously driven light-emitting channel are adjacent to each other so that the sensing area is appropriately complemented.

[0172] 30 and 31 are diagrams for explaining light intensity correction accompanying light emission from multiple channels, and Fig. 32 is a diagram showing an example of a system configuration related to light intensity correction.

[0173] In this example, a block scan drive system is assumed. Block scan refers to a system in which the SPAD array and laser channels simultaneously project and receive light onto multiple discrete local regions (blocks) on the xy plane. The combination patterns of light-emitting channels that simultaneously project light are stored in the light-emitting channel pattern storage unit 59.

[0174] In the "block of interest" in Figure 30, before overlapping with the laser 35a associated with itself, it overlaps with the lasers 35a for multiple other blocks. This corresponds to the expansion of the light emission channel described above. However, because the blocks are discrete, light intensity correction according to the position in the z direction is necessary (see Figure 31).

[0175] For example, the feature acquisition unit 52 acquires, as the light reception data for each light reception channel, light reception data for the light emitted by the light emission channel (own channel) corresponding to this light reception channel, and light reception data for the light emitted by one or more other light emission channels (other channels) that emit light closer to the near side within the field of view of the light reception channel (own channel) than this light emission channel (own channel).

[0176] Because the Z waveform changes compared to when only the channel itself is illuminated, it is preferable to acquire and evaluate the feature quantities using appropriate correction conditions. For example, based on the geometric arrangement (hardware specifications related to laser arrangement) shown in FIG. 30 , it is possible to pre-calculate the overlapping points of light rays in local xyz space (circled areas in FIG. 30 ) and the correction ratios (see the lower diagram in FIG. 31 ). The pre-calculated information is registered in the light intensity correction LUT 56 provided in the fusion system 10. The light intensity correction LUT 56 stores information on the correction ratios corresponding to an increase in the number of light-receiving channels, in association with the hardware specifications of the LiDAR sensor 31.

[0177] Based on the hardware specifications acquired by register input, the light intensity correction LUT 56 outputs information on the corresponding correction ratio to downstream processing blocks such as the feature acquisition unit 52, scattering detection unit 11, and scattering model correction unit 53. Each processing block performs multiplication of the correction ratio locally in x, y, and z to acquire and determine the feature amounts related to luminance.

[0178] Examples of correction targets include peak height and cumulative power in the feature acquisition unit 52, peak height, cumulative power, and effective brightness bin in the scattering detection unit 11, and the waveform of the estimated model in the scattering model correction unit 53.

[0179] 3-9. Reduction of Dead Areas by Expanding Light Receiving Channels FIGS. 33 and 34 are diagrams for explaining reduction of dead areas DA by expanding light receiving channels.

[0180] As described above, in normal LiDAR measurement, only one corresponding light-receiving channel is driven for each light-emitting channel. However, in this example, multiple light-receiving channels are simultaneously driven for one light-emitting channel. Another light-receiving channel facing closer to the foreground than the channel in question is simultaneously driven, and the RAW histogram for the foreground section of the channel in question (the foreground section) is replaced with the RAW histogram for the foreground section of the other channel (see FIG. 34). The RAW histogram for the background section (the +z side) of the channel in question (the background section) is not used in the analysis.

[0181] For example, the feature acquisition unit 52 acquires, as light reception data of the irradiated light from each light emission channel, light reception data of the light reception channel (own channel) corresponding to the light emission channel, and light reception data of one or more other light reception channels (other channels) that have a field of view on the near side within the irradiation range of the light emission channel than this light reception channel (own channel).

[0182] The feature amount acquisition unit 52 acquires light reception data for the background section from a light reception channel (own channel) corresponding to the light emission channel. The feature amount acquisition unit 52 acquires light reception data for the foreground section from one or more other light reception channels (other channels). The feature amount acquisition unit 52 combines the light reception data for the foreground section and the light reception data for the background section to acquire the light reception data that is the subject of acquisition of the Z waveform feature amount WF.

[0183] As shown in Figure 34, the sensor field of view of the other light receiving channel is closer to the foreground than the own channel. Therefore, the blind area DA on the foreground side is smaller. By combining the foreground side data section of the own channel with the foreground side RAW histogram data of the other channel, the blind area DA is reduced. The system developer can freely set the extent of the data section to be replaced. To deal with discontinuities in the data, it is preferable to perform light intensity correction as shown in Figure 31.

[0184] The number of simultaneously driven separate channels may be one or more. In the example of Figure 33, two light receiving channels whose sensor field of view ranges are adjacent to each other are driven simultaneously. The sensor field of view range means the angle of view at which light can be received. The more the number of simultaneously driven light receiving channels increases, the more the blind area DA is reduced. It is preferable that the sensor field of view ranges of each simultaneously driven light receiving channel are adjacent to each other so that the sensing area is appropriately complemented.

[0185] [3-10. Improvement of Estimation Accuracy of Parallax Pattern Using History Information] FIG. 35 is a diagram for explaining improvement of estimation accuracy of the parallax pattern PT using history information.

[0186] The fusion system 10 stores information used or estimated in the internal estimation process of the disparity pattern PT in the history information storage unit 57. The information to be stored includes, for example, the disparity amount SL_diff and RAW histogram data shown in FIG. 17. The disparity amount SL_diff that gives the maximum value of "Conf_pred" in equation (12) is stored. The disparity pattern acquisition unit 51 improves the estimation accuracy of the disparity pattern PT according to the following method.

[0187] <Method 1> The disparity pattern acquisition unit 51 averages the disparity amounts SL_diff for the most recent frames and acquires the average value as the estimated value of the disparity amount SL_diff for the current frame. The averaging may include an estimated value estimated using only the RAW histogram of the current frame. This allows the disparity pattern acquisition unit 51 to acquire average specifications based on the estimation history of the hardware specifications as the hardware specifications of the LiDAR sensor 31.

[0188] The disparity pattern acquisition unit 51 can reduce the value of the estimation reliability "Conf_pred" according to the variation σ_SL of the disparity amount SL_diff to be averaged. For example, the value of the estimation reliability "Conf_pred" after correction complies with the following formula (39). Corrected Conf_pred=Uncorrected Conf_pred-σ_SL / Averaged SL_diff (39)

[0189] When the LiDAR sensor 31 moves, the signal of the scattered component to be detected becomes steady. On the other hand, the signal of the object approaches the near side, so the estimated value fluctuates greatly. By reflecting such fluctuations in the estimated value (variation σ_SL) in the estimation reliability "Conf_pred," the estimation accuracy of the disparity pattern PT can be improved.

[0190] <Method 2> The disparity pattern acquisition unit 51 can estimate the hardware specifications of the LiDAR sensor 31 based on the cumulative value of the most recent RAW histogram. For example, the disparity pattern acquisition unit 51 adds up RAW histograms of the foreground section (z<Z_lim) of past frames and acquires the pattern feature PF from the added RAW histograms for multiple frames. This suppresses a decrease in the estimation accuracy of the disparity pattern PT due to noise.

[0191] 3-11. Improvement in Scattering Detection Accuracy by Pre-Extraction of Clear Peaks FIGS. 36 to 38 are diagrams for explaining improvement in scattering detection accuracy by pre-extraction of clear peaks.

[0192] The scattering detection unit 11 can determine the presence or absence of scattering based on a clear peak (clear peak) that is presumed to be the signal from the subject SB. For example, if another object exists behind the subject SB (on the +z side), the reflected light from the other object is blocked by the subject SB and does not reach the light receiving sensor 36a (occlusion). Therefore, the signal behind the clear peak is considered to be noise. Since the area in front of the clear peak (on the -z side) is considered to be a space with no objects, if a signal with effective brightness exists in front of the clear peak, it can be presumed that the signal is due to scattering.

[0193] For example, the fusion system 10 of this embodiment includes a clear peak pre-extraction unit 58. The clear peak pre-extraction unit 58 extracts peaks representing objects other than scattering media from the RAW histogram as clear peaks. The disparity pattern acquisition unit 51 estimates the luminance distribution on the closer side of the clear peak as the luminance distribution due to scattering. The disparity pattern acquisition unit 51 can exclude the luminance distribution on the farther side of the clear peak from the target for matching with the disparity pattern PT.

[0194] For example, scattering detection based on distinct peaks can be performed in the following manner.

[0195] First, the clear peak pre-extraction unit 58 searches for a peak of the Z waveform in the z direction for each pixel (X, Y) at a fixed interval, and extracts a peak (clear peak) that is brighter than the surrounding bins. A method for extracting peak heights from a Z waveform containing scattered components is, for example, to obtain the brightness difference between the peak bin (argmax in a local z-direction interval) and the position of a bin offset in the z direction based on the following formula (40): Peak height = L(X, Y, Zpeak) - L(X, Y, Zpeak + Zoffset) (40)

[0196] When the peak height calculated by equation (40) is equal to or greater than a threshold, the clear peak pre-extraction unit 58 extracts the peak that is the subject of the calculation as a clear peak. The threshold can be set, for example, based on the following equation (41). In equation (41), "σ" is the noise standard deviation, and "C_noise" is a coefficient. Threshold=C_noise×σ (41)

[0197] For example, the disparity pattern acquisition unit 51 excludes the RAW histogram of a pixel having a clear peak in the foreground section (z<Z_lim) from the calculation target for the internal estimation. This is because a clear peak is likely to represent an object, and may be adversely affected by occlusion behind the object, etc., and thus may affect the calculation of the internal estimation process (see FIG. 37 ).

[0198] The scattering detection unit 11 can add the percentage of pixels that satisfy the following conditions (condition I, condition II: see FIG. 38) to the conditions for scattering detection. Alternatively, the scattering detection unit 11 can determine that scattering has been detected if the percentage of pixels that satisfy the following two conditions is equal to or greater than a certain level. Condition I: A clear peak exists behind the foreground (z>Z_lim). Condition II: There is sufficient signal information in the foreground.

[0199] Condition II can be determined by setting a threshold value for the Z waveform feature amount WF (peak height, cumulative power, number of effective brightness bins, etc.). Condition I can be determined by, for example, weighting "Score6" in the following formula (42) and adding it to the consistency score "DetScore" in formula (20). Score6 = (pixels with clear peaks) / (total pixels) (42)

[0200] The scattering rate / reliability calculation unit 18 removes echoes that are located before a clear peak in a pixel where the clear peak exists, while protecting the clear peak as the object SB (see FIG. 38).

[0201] For example, the scattering elimination unit 13 selectively uses the LiDAR data LD in a section (non-scattering determination section) other than the scattering determination section SD to extract a clear peak (subject depth DP). The scattering elimination unit 13 detects a clear peak based on condition 3 described in FIG.

[0202] That is, the scattering rate / reliability calculation unit 18 calculates the reliability Conf of the local scattering rate α(Z). α The condition that (Z) is less than the threshold (Conf α (Z) <Conf αｔｈｒｅｓWhen condition 3) is satisfied for the first time, the region in the sensing space up to the position in the z direction where that condition is satisfied for the first time is detected as the scattering determination section SD (see the bottom diagram in FIG. 11 ). The scattering exclusion unit 13 selectively uses the LiDAR data LD of sections other than the scattering determination section SD (non-scattering determination sections) to extract a clear peak (subject depth DP).

[0203] For example, the confidence level Conf α (Z) is calculated according to the formulas (7) and (9). The scattering rate and reliability calculation unit 18 calculates the reliability Conf α (Z) is set to 1 (> Conf) just before the clear peak only when a clear peak is found. αｔｈｒｅｓ ) and 0 (<Conf αｔｈｒｅｓ ) This turns scattering detection ON / OFF.

[0204] 4. Example of Hardware Configuration FIG. 39 is a diagram showing an example of the hardware configuration of the fusion system 10. As shown in FIG.

[0205] The information processing of the fusion system 10 is realized by, for example, a computer 1000. The computer 1000 has a central processing unit (CPU) 1100, a random access memory (RAM) 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0206] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0207] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0208] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0209] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0210] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, or semiconductor memory.

[0211] For example, when the computer 1000 functions as the fusion system 10 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0212] [5. Effects] The fusion system 10 includes a parallax pattern acquisition unit 51 and a scattering detection unit 11. The parallax pattern acquisition unit 51 acquires, as a parallax pattern PT, a luminance distribution in the depth direction and a parallax direction orthogonal to the depth direction, which is assumed in a scattering environment. The scattering detection unit 11 acquires, as a measured pattern MP, a luminance distribution in the depth direction and the parallax direction indicated by a RAW histogram of the LiDAR sensor 31. The scattering detection unit 11 compares the measured pattern MP with the parallax pattern PT to determine the presence or absence of scattering in the sensing space. In the information processing method disclosed herein, the processing of the fusion system 10 is executed by a computer 1000. A computer-readable non-transitory storage medium disclosed herein stores a program that causes the computer 1000 to implement the processing of the fusion system 10.

[0213] According to this configuration, the presence or absence of scattering is detected based on the luminance distribution (measured pattern MP) in the depth direction and the parallax direction. In a scattering environment, a unique luminance distribution is observed in the parallax direction due to the parallax between the laser 35a and the light receiving sensor 36a. In the present disclosure, the presence or absence of scattering is detected by modeling this luminance distribution specific to the parallax direction. Therefore, the presence or absence of scattering can be detected more accurately than when a one-dimensional luminance distribution (Z waveform) in only the depth direction is used.

[0214] The fusion system 10 has a feature amount acquisition unit 52. The feature amount acquisition unit 52 acquires a waveform in the depth direction indicated by the RAW histogram as a Z waveform. The feature amount acquisition unit 52 acquires the feature amount of the Z waveform as a Z waveform feature amount WF. The scattering detection unit 11 scores the consistency between the Z waveform and the parallax pattern PT based on the Z waveform feature amount WF. The scattering detection unit 11 determines the presence or absence of scattering based on the calculated consistency score.

[0215] According to this configuration, the comparison between the measured pattern MP and the parallax pattern PT is performed with high accuracy.

[0216] The feature amount acquiring unit 52 acquires a dead zone DZ where sensing is impossible based on the parallax pattern PT. The feature amount acquiring unit 52 excludes the raw histogram data of the dead zone DZ from the acquisition target for the Z waveform feature amount WF.

[0217] According to this configuration, unnecessary calculations can be omitted and the Z waveform feature value WF can be obtained efficiently and accurately.

[0218] The disparity pattern acquisition unit 51 acquires bins having valid luminance values ​​from the RAW histogram as valid luminance bins. The disparity pattern acquisition unit 51 acquires feature quantities related to the spatial arrangement of the valid luminance bins in the depth direction and the parallax direction as pattern feature quantities PF. The disparity pattern acquisition unit 51 estimates hardware specifications related to the laser arrangement of the LiDAR sensor 31 based on the pattern feature quantities PF. The disparity pattern acquisition unit 51 acquires luminance distributions in the depth direction and the parallax direction estimated from the hardware specifications as a disparity pattern PT.

[0219] According to this configuration, a highly accurate parallax pattern PT can be obtained.

[0220] The disparity pattern acquisition unit 51 acquires, from the depth direction waveform shown by the RAW histogram, the bin closest to the foreground where effective luminance values ​​begin to appear as the effective luminance start bin. The disparity pattern acquisition unit 51 acquires, as the pattern boundary line BL, the line drawn by the effective luminance start bin in a plane including the depth direction and the disparity direction. The disparity pattern acquisition unit 51 acquires, as the pattern feature amount PF, one or more singular points SP of the pattern boundary line BL.

[0221] According to this configuration, the pattern feature PF that properly reflects the hardware specifications is acquired.

[0222] The parallax pattern acquisition unit 51 acquires the arrangement period of the singular points SP in the parallax direction. Based on the arrangement period of the singular points SP and the position of each singular point SP, the parallax pattern acquisition unit 51 estimates the number of lasers 35 a, the parallax amount and parallax direction of each laser 35 a, the number of light-emitting channels per laser, and the light-receiving channels that are driven simultaneously with each light-emitting channel. The parallax pattern acquisition unit 51 acquires the estimated information as hardware specifications.

[0223] According to this configuration, a highly accurate parallax pattern PT is obtained based on a specific hardware structure.

[0224] The scattering detection unit 11 scores the consistency between the Z waveform in the vicinity of the pattern boundary line BL and the parallax pattern PT.

[0225] According to this configuration, scattering determination can be performed with high accuracy based on the luminance distribution that appears near the pattern boundary line BL and is specific to the scattering environment.

[0226] The scattering detection unit 11 acquires the reliability of the pattern boundary line BL, which is scored based on the position of the pattern boundary line BL assumed in a scattering environment and the luminance distribution in the vicinity of the pattern boundary line BL. The scattering detection unit 11 determines the presence or absence of scattering based on a consistency score that takes into account the score related to the reliability of the pattern boundary line BL.

[0227] According to this configuration, scattering determination can be performed with high accuracy.

[0228] The scattering detection unit 11 acquires a correct disparity pattern PT that is pre-registered in a correct pattern LUT based on the hardware specifications of the LiDAR sensor 31. The scattering detection unit 11 determines the presence or absence of scattering based on a consistency score that takes into account a reliability score based on the difference between the disparity pattern PT estimated from the RAW histogram and the pre-registered disparity pattern PT.

[0229] According to this configuration, scattering determination can be performed with high accuracy.

[0230] The parallax pattern acquisition unit 51 acquires average specifications based on the estimated history of hardware specifications as the hardware specifications of the LiDAR sensor 31.

[0231] According to this configuration, scattering determination can be performed with high accuracy while suppressing variations between measurements.

[0232] The parallax pattern acquisition unit 51 estimates the hardware specifications of the LiDAR sensor 31 based on the cumulative value of the most recent RAW histogram.

[0233] This configuration prevents a decrease in accuracy due to noise.

[0234] The parallax pattern acquisition unit 51 acquires hardware specifications related to laser placement from the LiDAR sensor 31. The parallax pattern acquisition unit 51 acquires a parallax pattern PT corresponding to the hardware specifications of the LiDAR sensor 31 from a correct pattern LUT that records a correct parallax pattern PT for each hardware specification.

[0235] According to this configuration, an accurate parallax pattern PT is obtained.

[0236] The fusion system 10 includes a remaining power estimation unit 17 and a scattering model correction unit 53. The remaining power estimation unit 17 estimates a scattering model according to the Z waveform feature value WF. The scattering model correction unit 53 calculates the gain of the scattering model based on the parallax pattern PT. The scattering model correction unit 53 acquires the scattering model corrected by the gain as a corrected scattering model.

[0237] According to this configuration, a corrected scattering model is acquired in which the attenuation of luminance due to parallax is corrected. By using the corrected scattering model, it is possible to accurately recognize the scattering environment.

[0238] The feature acquisition unit 52 acquires, as the light reception data for each light reception channel, light reception data for the light emitted by the light emission channel (own channel) corresponding to the light reception channel, and light reception data for the light emitted by one or more other light emission channels (other channels) that emit light closer to the near field within the field of view of the light reception channel than the light emission channel (own channel).

[0239] According to this configuration, the dead zone DZ is reduced, and scattering can be determined with high accuracy.

[0240] The fusion system 10 includes a light intensity correction LUT 56. The light intensity correction LUT 56 stores information on a correction ratio corresponding to an increase in the number of light emission channels that are the subject of light reception.

[0241] This configuration enables accurate determination of scattering.

[0242] The feature acquisition unit 52 acquires, as light reception data of the irradiated light from each light emission channel, light reception data of the light reception channel (own channel) corresponding to the light emission channel, and light reception data of one or more other light reception channels (other channels) that have a field of view on the near side within the irradiation range of the light emission channel than the light reception channel (own channel).

[0243] According to this configuration, the dead zone DZ is reduced, and scattering can be determined with high accuracy.

[0244] The feature amount acquisition unit 52 acquires light reception data for the background section from a light reception channel (own channel) corresponding to the light emission channel. The feature amount acquisition unit 52 acquires light reception data for the foreground section from one or more other light reception channels (other channels). The feature amount acquisition unit 52 combines the light reception data for the foreground section and the light reception data for the background section to acquire the light reception data that is the subject of acquisition of the Z waveform feature amount WF.

[0245] According to this configuration, the dead zone DZ is reduced, and scattering can be determined with high accuracy.

[0246] The parallax pattern acquisition unit 51 acquires a parallax pattern PT for each laser 35 a based on the hardware specifications of the LiDAR sensor 31 including the plurality of lasers 35 a. The scattering detection unit 11 compares the plurality of parallax patterns PT corresponding to the plurality of lasers 35 a with the actual measurement pattern MP to determine whether scattering is present or not.

[0247] According to this configuration, the number of types of parallax patterns PT increases, thereby improving the accuracy of scattering determination.

[0248] The scattering detection unit 11 acquires a plurality of measured patterns MP obtained by shifting the irradiation timing of the plurality of lasers 35a. The scattering detection unit 11 compares the acquired plurality of measured patterns MP with the corresponding parallax patterns PT. The scattering detection unit 11 combines the individual comparison results obtained for each parallax pattern PT to determine whether scattering is present or not.

[0249] According to this configuration, the number of types of parallax patterns PT increases, thereby improving the accuracy of scattering determination.

[0250] The fusion system 10 includes an ambient light acquisition unit 55. The ambient light acquisition unit 55 acquires the luminance distribution caused by the ambient light EL indicated by the RAW histogram as an ambient light image EI. The scattering detection unit 11 compares the ambient light image EI with the luminance distribution caused by the ambient light expected in a scattering environment to determine whether scattering exists.

[0251] According to this configuration, the presence or absence of scattering can be determined based on the ambient light EL.

[0252] The fusion system 10 includes a clear peak pre-extraction unit 58. The clear peak pre-extraction unit 58 extracts peaks representing objects other than scattering media from the RAW histogram as clear peaks. The parallax pattern acquisition unit 51 estimates the luminance distribution on the near side of the clear peak as the luminance distribution due to scattering.

[0253] This configuration makes it possible to easily identify data regions that exhibit scattering.

[0254] The parallax pattern acquisition unit 51 excludes the luminance distribution on the distant side of the clear peak from the objects to be compared with the parallax pattern PT.

[0255] According to this configuration, by omitting calculations for distant view regions that are considered to be occluded, scattering determination can be performed efficiently and accurately.

[0256] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0257] [Additional Notes] The present technology may also be configured as follows: (1) An information processing device comprising: a disparity pattern acquisition unit that acquires, as a disparity pattern, a luminance distribution in a depth direction and a disparity direction orthogonal to the depth direction, which is assumed in a scattering environment; and a scattering detection unit that acquires, as a measured pattern, a luminance distribution in the depth direction and the disparity direction indicated by a RAW histogram of a LiDAR sensor, and compares the measured pattern with the disparity pattern to determine the presence or absence of scattering in a sensing space. (2) The information processing device according to (1), further comprising: a feature acquisition unit that acquires, as a Z waveform, a waveform in the depth direction indicated by the RAW histogram, and acquires a feature of the Z waveform as a Z waveform feature, wherein the scattering detection unit scores the consistency between the Z waveform and the disparity pattern based on the Z waveform feature, and determines the presence or absence of scattering based on the calculated consistency score. (3) The information processing device according to (2), wherein the feature amount acquisition unit acquires a blind section where sensing is impossible based on the disparity pattern, and excludes data of the RAW histogram of the blind section from targets for acquiring the Z waveform feature amount. (4) The information processing device according to (2) or (3), wherein the disparity pattern acquisition unit acquires bins having valid brightness values ​​from the RAW histogram as effective brightness bins, acquires feature amounts related to a spatial arrangement of the effective brightness bins in the depth direction and the disparity direction as pattern feature amounts, estimates hardware specifications related to a laser arrangement of the LiDAR sensor based on the pattern feature amounts, and acquires the brightness distribution in the depth direction and the disparity direction assumed from the hardware specification as the disparity pattern. (5) The information processing device according to (4), wherein the disparity pattern acquisition unit acquires, from the depth direction waveform shown by the RAW histogram, a bin on the nearest side where effective luminance values ​​begin to appear as an effective luminance start bin, acquires, as a pattern boundary line, a line drawn by the effective luminance start bin in a plane including the depth direction and the disparity direction, and acquires, as the pattern feature, one or more singular points of the pattern boundary line.(6) The information processing device according to (5), wherein the parallax pattern acquisition unit acquires an arrangement period of the singular points in the parallax direction, and estimates the number of lasers, the parallax amount and the parallax direction of each laser, the number of light-emitting channels per laser, and the light-receiving channels driven simultaneously with each light-emitting channel based on the arrangement period of the singular points and the position of each singular point, and acquires the estimated information as the hardware specifications. (7) The information processing device according to (5) or (6), wherein the scattering detection unit scores the consistency between the Z waveform in the vicinity of the pattern boundary line and the parallax pattern. (8) The information processing device according to any one of (5) to (7), wherein the scattering detection unit acquires reliability of the pattern boundary line scored based on the position of the pattern boundary line expected in the scattering environment and the luminance distribution in the vicinity of the pattern boundary line, and determines the presence or absence of scattering based on the consistency score taking into account the score related to the reliability of the pattern boundary line. (9) The information processing device according to any one of (5) to (8), wherein the scattering detection unit acquires the correct disparity pattern pre-registered in a correct pattern LUT based on hardware specifications of the LiDAR sensor, and determines the presence or absence of scattering based on a consistency score taking into account a reliability score based on a difference between the disparity pattern estimated from the RAW histogram and the pre-registered disparity pattern. (10) The information processing device according to any one of (4) to (9), wherein the disparity pattern acquisition unit acquires average specifications based on an estimation history of the hardware specifications as the hardware specifications of the LiDAR sensor. (11) The information processing device according to any one of (4) to (9), wherein the disparity pattern acquisition unit estimates the hardware specifications of the LiDAR sensor based on a cumulative value of the most recent RAW histogram.(12) The information processing device according to (2) or (3), wherein the disparity pattern acquisition unit acquires hardware specifications related to laser placement from the LiDAR sensor, and acquires the disparity pattern according to the hardware specifications of the LiDAR sensor from a correct pattern LUT in which the correct disparity pattern for each of the hardware specifications is recorded. (13) The information processing device according to any one of (2) to (12), comprising: a residual power estimation unit that estimates a scattering model according to the Z waveform feature amount; and a scattering model correction unit that calculates a gain of the scattering model based on the disparity pattern and acquires the scattering model corrected by the gain as a corrected scattering model. (14) The information processing device according to (13), wherein the feature acquisition unit acquires, as the light reception data of each light reception channel, light reception data of irradiation light of an light emission channel corresponding to the light reception channel, and light reception data of irradiation light of one or more other light emission channels that irradiate light closer to the near view side within the field of view of the light reception channel than the light emission channel. (15) The information processing device according to (14), further comprising a light intensity correction LUT that stores information on a correction ratio corresponding to an increase in the light-emitting channel that is the target of light reception. (16) The information processing device according to (13), further comprising: the feature acquisition unit acquires, as light reception data of light irradiated from each light-emitting channel, light reception data of a light-receiving channel corresponding to the light-emitting channel, and light reception data of one or more other light-receiving channels that have a field of view on the foreground side within the illumination range of the light-emitting channel than the light-receiving channel. (17) The information processing device according to (16), further comprising: the feature acquisition unit acquires light reception data of a distant view section from the light-receiving channel corresponding to the light-emitting channel; acquires light reception data of a near view section from the other one or more light-receiving channels; and combines the light reception data of the near view section with the light reception data of the distant view section to acquire light reception data that is the target of acquisition of the Z waveform feature.(18) The information processing device according to any one of (1) to (17), wherein the parallax pattern acquisition unit acquires the parallax pattern for each laser based on hardware specifications of the LiDAR sensor including a plurality of lasers, and the scattering detection unit determines the presence or absence of scattering by comparing a plurality of parallax patterns corresponding to the plurality of lasers with the measured pattern. (19) The information processing device according to (18), wherein the scattering detection unit acquires a plurality of measured patterns obtained by shifting the irradiation timing of the plurality of lasers, compares the acquired plurality of measured patterns with the corresponding parallax patterns, and determines the presence or absence of scattering by integrating individual comparison results obtained for the parallax patterns. (20) The information processing device according to any one of (1) to (19), further comprising an ambient light acquisition unit that acquires a luminance distribution caused by ambient light indicated by the RAW histogram as an ambient light image, and the scattering detection unit determines the presence or absence of scattering by comparing the ambient light image with a luminance distribution caused by ambient light expected in the scattering environment. (21) The information processing device according to any one of (1) to (20), further comprising: a clear peak pre-extraction unit that extracts a peak indicating an object other than a scattering medium from the RAW histogram as a clear peak, and the disparity pattern acquisition unit estimates the luminance distribution on the closer side of the clear peak as the luminance distribution due to scattering. (22) The information processing device according to (21), further comprising: the disparity pattern acquisition unit excludes the luminance distribution on the farther side of the clear peak from a target for matching with the disparity pattern. (23) An information processing method executed by a computer, comprising: acquiring, as a disparity pattern, a luminance distribution in a depth direction and a disparity direction orthogonal to the depth direction that is assumed in a scattering environment; acquiring, as a disparity pattern, the luminance distribution in the depth direction and the disparity direction indicated by a RAW histogram of a LiDAR sensor; and comparing the measured pattern with the disparity pattern to determine the presence or absence of scattering in a sensing space.(24) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following: acquire, as a disparity pattern, a luminance distribution in a depth direction and a disparity direction perpendicular to the depth direction that is assumed in a scattering environment; acquire, as a measured pattern, a luminance distribution in the depth direction and the disparity direction shown by a RAW histogram of a LiDAR sensor; and compare the measured pattern with the disparity pattern to determine the presence or absence of scattering in a sensing space.

[0258] 10 Fusion system (information processing device) 11 Scattering detection unit 17 Residual power estimation unit 31 LiDAR sensor 35a Laser 36a Light receiving sensor 51 Parallax pattern acquisition unit 52 Feature acquisition unit 53 Scattering model correction unit 55 Ambient light acquisition unit 56 Light intensity correction LUT 58 Clear peak pre-extraction unit BL Pattern boundary line DZ Insensitive section EI Ambient light image EL Ambient light MP Measured pattern PF Pattern feature amount PT Parallax pattern SP Singular point WF Z waveform feature amount

Claims

1. An information processing device having: a disparity pattern acquisition unit that acquires, as a disparity pattern, the brightness distribution in the depth direction and the disparity direction perpendicular to the depth direction that is assumed in a scattering environment; and a scattering detection unit that acquires, as a measured pattern, the brightness distribution in the depth direction and the disparity direction shown by a RAW histogram of a LiDAR sensor, and compares the measured pattern with the disparity pattern to determine the presence or absence of scattering in a sensing space.

2. The information processing device of claim 1, further comprising a feature acquisition unit that acquires the depth direction waveform indicated by the RAW histogram as a Z waveform and acquires the feature of the Z waveform as a Z waveform feature, and the scattering detection unit scores the consistency between the Z waveform and the disparity pattern based on the Z waveform feature and determines the presence or absence of scattering based on the calculated consistency score.

3. The information processing device according to claim 2, wherein the feature acquisition unit acquires an insensitive section where sensing is not possible based on the parallax pattern, and excludes the RAW histogram data of the insensitive section from the acquisition target of the Z waveform feature.

4. The information processing device of claim 2, wherein the disparity pattern acquisition unit acquires bins having valid brightness values ​​from the RAW histogram as effective brightness bins, acquires features related to the spatial arrangement of the effective brightness bins in the depth direction and the disparity direction as pattern features, estimates hardware specifications related to the laser arrangement of the LiDAR sensor based on the pattern features, and acquires the brightness distribution in the depth direction and the disparity direction assumed from the hardware specifications as the disparity pattern.

5. The information processing device according to claim 4, wherein the disparity pattern acquisition unit acquires, from the depth direction waveform shown by the RAW histogram, the bin closest to the nearest view where effective luminance values ​​begin to appear as an effective luminance start bin, acquires, as a pattern boundary line, a line drawn by the effective luminance start bin in a plane including the depth direction and the disparity direction, and acquires, as the pattern feature, one or more singular points that the pattern boundary line has.

6. The information processing device according to claim 4, wherein the disparity pattern acquisition unit acquires average specifications based on an estimation history of the hardware specifications as the hardware specifications of the LiDAR sensor.

7. The information processing device according to claim 4, wherein the disparity pattern acquisition unit estimates the hardware specifications of the LiDAR sensor based on a cumulative value of the most recent RAW histogram.

8. The information processing device described in claim 2, wherein the disparity pattern acquisition unit acquires hardware specifications regarding laser placement from the LiDAR sensor, and acquires the disparity pattern corresponding to the hardware specifications of the LiDAR sensor from a correct pattern LUT that records the correct disparity pattern for each of the hardware specifications.

9. The information processing device according to claim 2, comprising: a residual power estimation unit that estimates a scattering model according to the Z waveform feature amount; and a scattering model correction unit that calculates a gain of the scattering model based on the disparity pattern and acquires the scattering model corrected by the gain as a corrected scattering model.

10. The information processing device described in claim 9, wherein the feature acquisition unit acquires, as the light reception data for each light reception channel, light reception data for the light emitted by the light emission channel corresponding to the light reception channel, and light reception data for the light emitted by one or more other light emission channels that emit light closer to the near field within the field of view of the light reception channel than the light emission channel.

11. The information processing device according to claim 10, further comprising a light intensity correction LUT that stores information on a correction ratio corresponding to an increase in the light emission channel that is the light receiving target.

12. The information processing device described in claim 9, wherein the feature acquisition unit acquires, as light reception data of the light irradiated from each light emitting channel, light reception data of a light receiving channel corresponding to the light emitting channel, and light reception data of one or more other light receiving channels having a field of view on the near side within the irradiation range of the light emitting channel than the light receiving channel.

13. The information processing device described in claim 12, wherein the feature acquisition unit acquires light reception data for the background section from the light reception channel corresponding to the light emission channel, acquires light reception data for the foreground section from the other one or more light reception channels, and combines the light reception data for the foreground section with the light reception data for the background section to acquire the light reception data that is the subject of acquisition of the Z waveform feature.

14. The information processing device described in claim 1, wherein the parallax pattern acquisition unit acquires the parallax pattern for each laser based on hardware specifications of the LiDAR sensor including multiple lasers, and the scattering detection unit compares multiple parallax patterns corresponding to the multiple lasers with the actual measured pattern to determine whether scattering is present or not.

15. The information processing device according to claim 14, wherein the scattering detection unit acquires a plurality of measured patterns obtained by shifting the irradiation timing of the plurality of lasers, compares the acquired measured patterns with the corresponding disparity patterns, and integrates the individual comparison results obtained for each disparity pattern to determine whether or not scattering is present.

16. An information processing device as described in claim 1, further comprising an ambient light acquisition unit that acquires the luminance distribution caused by ambient light indicated by the RAW histogram as an ambient light image, and wherein the scattering detection unit compares the ambient light image with the luminance distribution caused by the ambient light expected in the scattering environment to determine whether or not scattering is present.

17. An information processing device according to claim 1, further comprising a clear peak pre-extraction unit that extracts a peak representing an object other than a scattering medium from the RAW histogram as a clear peak, and the parallax pattern acquisition unit estimates the luminance distribution on the near side of the clear peak as the luminance distribution due to scattering.

18. The information processing device according to claim 17, wherein the disparity pattern acquisition unit excludes the luminance distribution on the distant side of the clear peak from the objects to be compared with the disparity pattern.

19. An information processing method executed by a computer, comprising: acquiring a luminance distribution in a depth direction and a parallax direction perpendicular to the depth direction, which is assumed in a scattering environment, as a parallax pattern; acquiring a luminance distribution in the depth direction and the parallax direction shown by a RAW histogram of a LiDAR sensor as a measured pattern; and comparing the measured pattern with the parallax pattern to determine the presence or absence of scattering in a sensing space.

20. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: acquire, as a disparity pattern, the brightness distribution in the depth direction and the disparity direction perpendicular to the depth direction that is expected in a scattering environment; acquire, as a measured pattern, the brightness distribution in the depth direction and the disparity direction shown by the RAW histogram of a LiDAR sensor; and compare the measured pattern with the disparity pattern to determine the presence or absence of scattering in the sensing space.

Images