Information processing apparatus, information processing method, and computer-readable non-transitory storage medium

Abstract

The information processing apparatus includes a scatter estimation unit and a fusion unit. A scattering rate estimation unit estimates a distribution of scattering rates in the sensing space from the LiDAR data. The fusion unit corrects the sensor data to be combined with the LiDAR data by the sensor fusion based on the distribution of the scattering rates prior to the sensor fusion.

Description

Technical Field

[0001] This invention relates to an information processing apparatus, an information processing method, and a computer-readable non-transitory storage medium. Background Technology

[0002] Systems that fuse data from LiDAR (Light Detection and Ranging) or stereo cameras to perform distance measurement and identification are known (fusion systems). Fusion systems are used in systems that require high-precision information, such as ADAS (Advanced Driver Assistance Systems).

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: JP 2019-124537 A Summary of the Invention

[0006] Technical issues

[0007] For safe autonomous driving, ADAS (Advanced Driver Assistance Systems) must be resilient to adverse weather conditions. However, in fusion systems, performance can sometimes deteriorate significantly in inclement weather. This is because the output signals of the front-end sensors (LiDAR or camera devices) are degraded by scattering from particles such as rain or fog, making it difficult to ensure the performance of the later-end fusion system. Radar (radio detection and ranging) is resistant to adverse weather, but its lower resolution makes it difficult to maintain overall system resolution while coping with inclement weather.

[0008] Therefore, this disclosure proposes an information processing apparatus, an information processing method, and a computer-readable non-transitory storage medium capable of constructing a fusion system with high resistance to severe weather.

[0009] Solution to the problem

[0010] According to this disclosure, an information processing apparatus is provided, comprising: a scattering rate estimation unit that estimates the distribution of scattering rate in a sensing space based on LiDAR data; and a fusion unit that, prior to sensor fusion, corrects sensor data combined with LiDAR data through sensor fusion based on the scattering rate distribution. According to this disclosure, an information processing method in which information processing is performed by a computer and a computer-readable non-transitory storage medium storing a program for causing the computer to perform information processing by the information processing apparatus are provided. Attached Figure Description

[0011] Figure 1 This is a diagram illustrating an example of a system equipped with a fusion system.

[0012] Figure 2It is a diagram used to illustrate performance degradation under severe weather conditions.

[0013] Figure 3 It is a diagram used to illustrate performance degradation under severe weather conditions.

[0014] Figure 4 This is a diagram illustrating a method for improving the performance of a fusion system as described in this disclosure.

[0015] Figure 5 This is a diagram illustrating a configuration example of an ADAS equipped with the fusion system of this disclosure.

[0016] Figure 6 This is a flowchart illustrating an overview of the processing of the fusion system.

[0017] Figure 7 This is a diagram showing the specific forms of the scattering rate estimation unit and the scattering exclusion unit.

[0018] Figure 8 This is a diagram used to illustrate the method of estimating local scattering rate based on residual power.

[0019] Figure 9 This is a diagram illustrating a method for estimating local scattering rate based on the blurring of the received light image.

[0020] Figure 10 This is a diagram used to illustrate a method for estimating local scattering rate based on the blurring of a received light image.

[0021] Figure 11 This is a diagram used to illustrate a method for estimating local scattering rate based on the blurring of a received light image.

[0022] Figure 12 This is a diagram used to illustrate a method for estimating local scattering rate based on the blurring of a received light image.

[0023] Figure 13 This is a diagram illustrating an example of a method for extracting the depth of a subject based on local scattering rate.

[0024] Figure 14 This is a diagram showing an example of data input to the LiDAR output unit.

[0025] Figure 15 This is a diagram showing an example of data output from the LiDAR output unit.

[0026] Figure 16 This is a diagram illustrating an example of a method for reducing waveform information.

[0027] Figure 17 This is a diagram illustrating an example of a method for reducing waveform information.

[0028] Figure 18 This is a diagram illustrating an example of a method for reducing waveform information.

[0029] Figure 19 This is a diagram showing the specific form of the fusion unit.

[0030] Figure 20 This is a diagram showing the modification of the estimation process for the local scattering rate.

[0031] Figure 21 This is a diagram illustrating an example of acquiring multiple LiDAR data with different laser powers.

[0032] Figure 22 This is a diagram illustrating an example of acquiring multiple LiDAR data with different laser powers.

[0033] Figure 23 This is a graph showing the changes in the output data of the LiDAR output unit.

[0034] Figure 24 This is a diagram illustrating the modifications made to the scattering removal process and the combination ratio acquisition process from the RGB image.

[0035] Figure 25 This is a diagram used to illustrate the re-rendering of a scattered RGB image.

[0036] Figure 26 This is a diagram illustrating an example of the hardware configuration of a fusion system. Detailed Implementation

[0037] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. In the embodiments described below, repeated descriptions are omitted by adding the same reference numerals to the same parts.

[0038] Note that the explanations will be given in the following order.

[0039] [1. Overview]

[0040] [1-1. Positioning of the Fusion System]

[0041] [1-2. Performance degradation under severe weather conditions]

[0042] [1-3. Improving performance through scattering rate distribution estimation and data recovery]

[0043] [2. The system for integrating the content of this disclosure]

[0044] [2-1. Processing Overview]

[0045] [2-2. Scattering Rate Estimation Unit / Scattering Exclusion Unit]

[0046] [2-2-1. Estimation of local scattering rate based on residual power]

[0047] [2-2-2. Estimation of blurry local scattering rate based on received light image]

[0048] [2-2-3. Subject Depth Extraction Based on Local Scattering Rate]

[0049] [2-3. LiDAR Output Unit]

[0050] [2-4. Fusion Unit]

[0051] [3. Revision]

[0052] [3-1. Feedback of corrected scattering rate]

[0053] [3-2. Acquisition of multiple LiDAR data with different laser powers]

[0054] [3-3. Changes in data format]

[0055] [3-4. Feedback on corrected depth]

[0056] [3-5. Re-rendering of scattering RGB images]

[0057] [4. Hardware Configuration Example]

[0058] [5. Effects]

[0059] [1. Overview]

[0060] [1-1. Positioning of the Fusion System]

[0061] Figure 1 This is a diagram illustrating an example of a system equipped with a fusion system.

[0062] Fusion systems are used in systems requiring high-precision information, such as ADAS. ADAS typically has a two-stage configuration consisting of a sensor array and a fusion system. The sensor array, serving as the front-end component, includes multiple ranging sensors such as cameras, LiDAR, and radar. The various sensor data detected by the sensor array are output to the fusion system, serving as the back-end component. The fusion system combines multiple sensor data through sensor fusion to generate highly accurate output information.

[0063] exist Figure 1 The example shown illustrates corrected depth, subject movement, and subject identification tags as output information. These types of output information are used for environmental awareness, position estimation, object tracking, and more. Vehicle control algorithms use this output information to guide the control of various devices.

[0064] [1-2. Performance degradation under severe weather conditions]

[0065] Figure 2 and Figure 3 It is a diagram used to illustrate performance degradation under severe weather conditions.

[0066] In sensors (optical sensors) such as cameras or LiDAR that detect light and convert it into electrical signals, significant signal degradation occurs at the time of sensor output due to scattering and absorption by rain, fog, etc. For example, in LiDAR measurements, the scattered light caused by scattering particles becomes greater than the reflected light from the object, making it impossible to detect the object. In camera measurements, because the scattering of ambient light superimposes in the depth direction, objects become increasingly blurry and discriminability degrades as they become more distant. Even by combining the degraded signals through sensor fusion, accurate output information cannot be obtained. Although radar is highly resistant to harsh weather conditions, if radar is used as the primary source, its relatively low resolution cannot guarantee sufficient performance.

[0067] For example, the scattering model of LiDAR can be represented by a formula similar to Equation (1) below. In Equation (1), P(z) represents the power (residual power) at a distance z. P0 represents the intensity of the laser at the time of laser irradiation (laser emission intensity). α(x, y, z) represents the local scattering rate at point (x, y, z). r represents the correction coefficient. (x, y, z) indicates the coordinates of a point in a three-dimensional orthogonal coordinate system with the laser optical axis of the LiDAR sensor as the z-axis.

[0068]

[0069] In the following text, the local scattering rate will be referred to as the "local scattering rate". The local scattering rate α is a parameter indicating the concentration of scattering particles at each location. In this disclosure, the parameter indicating the concentration of scattering particles is defined as "scattering rate", but similar parameters include "scattering coefficient", "transmittance", etc. Basically, any parameter indicating concentration is included in the concept of "scattering rate".

[0070] In this disclosure, the scattering model indicated by Equation (1) is used, but various types of modeling, such as gamma distribution, are possible for the scattering model. The method of this disclosure is not limited to the specific scattering model indicated by Equation (1) and can be applied to all scattering models that include concentration parameters.

[0071] [1-3. Improving performance through scattering rate distribution estimation and data recovery]

[0072] Figure 4 This is a diagram illustrating a method for improving the performance of the fusion system according to the present disclosure.

[0073] As described above, the residual power P(z) decreases sharply with increasing distance z and local scattering rate α. This is because the signal from the subject is weakened by scattering, and on the other hand, the scattering signal is enhanced by scattering particles, resulting in very high noise. In this disclosure, this problem is addressed by estimating the scattering rate distribution and recovering data based on the estimation results.

[0074] For example, in Equation (1), the residual power P(z) is represented by the product of three terms. "Term A" indicates the distance-squared attenuation term. "Term B" indicates the reflectivity of the laser light through the scattering particles. "Term C" indicates the rate attenuation due to scattering as the light travels a distance z. For example, the scattering rate (local scattering rate) at each location is obtained by applying the actual measurement of the residual power P(z) to Equation (1). By correcting the LiDAR data based on the spatial distribution of the scattering rate, waveform information of the LiDAR with reduced noise caused by scattering (subject depth) is obtained.

[0075] Information about scattering rate can also be used to correct sensor data that is planned to be combined with LiDAR data LD through sensor fusion. In other words, information about scattering rate can also be used to correct sensor data in the pre-implementation stage of sensor fusion. Figure 4 An example is shown where an RGB image acquired by a camera device is used as sensor data to be combined. An RGB image is a color image that has color information such as red (R), green (G), and blue (B). RGB images are acquired, for example, using stereo imaging methods as images for distance measurement.

[0076] For example, terms A and B in equation (1) are specific to LiDAR (active sensor) and are not applicable to data from camera devices (passive sensor). On the other hand, term C indicates the attenuation factor corresponding to distance z and local scattering rate α, and is applicable to both LiDAR and camera device data. By correcting the RGB image based on information about the scattering rate, the RGB image in a state without noise caused by scattering can be estimated.

[0077] Information about scattering rate can also be used to estimate the reliability of sensor data. The combination ratio RA of sensor fusion can be adjusted based on the reliability of the sensor data (see [reference]). Figure 19 This can improve the accuracy of the output information. Note that in the following description, the RGB image of the camera device is shown as the sensor data to be combined with LiDAR data LD, but the sensor data is not limited to this. The same method can be applied to the combination of sensor data from other optical sensors such as IR sensors or EVS (event-based vision sensors).

[0078] [2. The system for integrating the content of this disclosure]

[0079] [2-1. Processing Overview]

[0080] Figure 5 This is a diagram illustrating a configuration example of ADAS1 equipped with the fusion system 10 of this disclosure. Figure 6 This is a flowchart illustrating an overview of the processing of the fusion system 10.

[0081] ADAS1 includes a sensor group 30 and a fusion system 10. The sensor group 30 includes a LiDAR sensor 31 and a camera device 32. For example, the LiDAR sensor 31 uses an image sensor that includes SPADs (single-photon avalanche diodes) in its pixels to perform distance measurement. The camera device 32 performs distance measurement using multiple RGB images from different viewpoints according to a stereo imaging method. The sensor group 30 may also include other optical sensors, such as IR sensors and EVS, in place of or in combination with the camera device 32.

[0082] The fusion system 10 is an information processing device that combines various sensor data acquired from the sensor group 30 and generates high-precision output information. For example, the fusion system 10 includes a scattering detection unit 11, a scattering rate estimation unit 12, a scattering exclusion unit 13, a LiDAR output unit 14, and a fusion unit 15.

[0083] Fusion System 10 Figure 6 The process shown in the diagram performs sensor fusion processing. First, the fusion system 10 acquires various sensor data related to the space to be sensed (sensing space) from the sensor group 30 (step S1). The sensor data includes LiDAR data LD (raw data) output from the LiDAR sensor 31 and RGB image IM (raw data) output from the camera device 32.

[0084] The scattering detection unit 11 determines whether scattering is present or absent by analyzing the LiDAR data LD (step S2). If scattering is absent (step S2: No), the fusion unit 15 combines the LiDAR data LD and the RGB image IM according to sensor fusion to generate a corrected depth DP. R (Step S6). The fusion system 10 is based on the RGB image IM and the corrected depth DP. R The system generates motion (MO) data for the subject and identification tag (RL) data. The fusion system 10 then identifies and tracks the subject based on the generated data (step S7).

[0085] In the presence of scattering (step S2: yes), the scattering rate estimation unit 12 calculates the local scattering rate LS in the sensing space based on equation (1).P (Step S3). Local scattering rate LS P This refers to the local scattering rate at each location (a point on three-dimensional coordinates) in the sensing space. The scattering rate estimation unit 12 estimates the distribution of the scattering rate in the sensing space based on the LiDAR data LD by calculating the scattering rate at each point in the sensing space. The scattering exclusion unit 13 excludes components caused by scattering from the LiDAR data LD based on the scattering rate distribution, and extracts the waveform information from which the effects of scattering have been excluded as the subject depth DP.

[0086] To reduce data volume, the LiDAR output unit 14 compresses the local scattering rate LS. P The data. The LiDAR output unit 14 will use the local scattering rate LS obtained through compression. B The data, along with the subject depth DP, is output to the fusion unit 15 (step S4).

[0087] Fusion unit 15 is based on local scattering rate LS B The data is used to estimate the degradation amount of the RGB image IM (step S5). The fusion unit 15 corrects the RGB image IM based on the estimated degradation amount and generates an RGB image without noise caused by scattering as the corrected image IM. R (Refer to Figure 19 ).

[0088] The fusion unit 15 calculates the corrected image IM based on the degradation of the RGB image IM. R The reliability of the points is calculated. The fusion unit 15 calculates the reliability of the subject depth DP based on the signal strength of the subject depth DP and the degradation of the RGB image IM. The fusion unit 15 determines the combination ratio RA of the points in sensor fusion based on the calculated reliability. The fusion unit 15 performs sensor fusion based on the determined combination ratio RA and outputs the combined depth information as the corrected depth DP. R (Step S6)

[0089] Fusion System 10 Based on Corrected Image IM R and corrected depth DP R The system generates motion (MO) data for the subject and identification tag (RL) data. The fusion system 10 then identifies and tracks the subject based on the generated data (step S7).

[0090] The fusion system 10 determines whether data exists for the next frame (step S8). If no data exists (step S8: No), the process ends. If data exists for the next frame (step S8: Yes), the process returns to step S1, and the fusion system 10 repeats the above process until no data exists.

[0091] [2-2. Scattering Rate Estimation Unit / Scattering Exclusion Unit]

[0092] Figure 7 This is a diagram showing the specific forms of the scattering rate estimation unit 12 and the scattering exclusion unit 13.

[0093] The scattering rate estimation unit 12 estimates information related to the scattering rate in the sensing space based on the sensing results of the LiDAR sensor 31. The LiDAR sensor 31 includes a LiDAR light projection unit 35 and a LiDAR light receiving unit 36. The LiDAR light projection unit 35 emits laser light towards the sensing space. The LiDAR light receiving unit 36 ​​receives the laser light irradiated from the object and reflected from it. The LiDAR light receiving unit 36 ​​converts the time (time of flight) for the laser light to be reflected back based on the speed of light into distance. The LiDAR light receiving unit 36 ​​sequentially outputs the brightness data of the received laser light to the scattering rate estimation unit 12.

[0094] The scattering rate estimation unit 12 acquires the brightness data sequentially output from the LiDAR light receiving unit 36 ​​as LiDAR data LD. The LiDAR data LD is a histogram of data where the horizontal axis represents distance and the vertical axis represents brightness. The scattering rate estimation unit 12 establishes a three-dimensional coordinate system in the sensing space, with the z-axis representing the distance direction, and acquires the brightness data of points in the sensing space from the LiDAR data LD. The scattering rate estimation unit 12 applies a scattering model to the three-dimensional brightness distribution to estimate the scattering rate distribution in the sensing space.

[0095] For example, the scattering rate estimation unit 12 includes a PSF acquisition unit 16, a residual power estimation unit 17, and a scattering rate / reliability calculation unit 18. The PSF acquisition unit 16 acquires information related to the blurring of the received light image caused by scattering based on LiDAR data LD. The residual power estimation unit 17 acquires information related to the scattered laser power (residual power P) based on LiDAR data LD and laser emission intensity P0. The scattering rate / reliability calculation unit 18 can estimate the distribution of the scattering rate based on the blurring of the received light image or the residual power P.

[0096] The following section details the estimation of the local scattering rate LS based on the blurring of the received light image. P Methods and estimation of local scattering rate based on residual power P (LS) P The method.

[0097] [2-2-1. Estimation of local scattering rate based on residual power]

[0098] Figure 8 It is used to illustrate the estimation of local scattering rate LS based on residual power P. P A diagram illustrating the method.

[0099] The residual power estimation unit 17 acquires the brightness data sequentially output from the LiDAR light receiving unit 36 ​​as LiDAR data LD. The LiDAR data LD is a histogram of data where the horizontal axis (z-axis) represents distance and the vertical axis represents brightness L. By sequentially acquiring brightness data from the LiDAR light receiving unit 36, the brightness waveform expands sequentially in the z-direction over time. Each time brightness L data is acquired, the residual power estimation unit 17 estimates the residual power P(z) at a distance z based on the known waveform history of laser emission intensity P0 and brightness L, according to the following equation (2).

[0100]

[0101] In equation (2), K(L(Z)) is a conversion function used to convert luminance L into power (light intensity). This conversion function, for example, reflects the nonlinear SPAD sensitivity characteristics.

[0102] The scattering rate / reliability calculation unit 18 estimates the local scattering rate α(Z) sequentially based on the brightness L(Z) and residual power P(Z) according to the following equations (3) and (4). The scattering rate / reliability calculation unit 18 obtains the scattering rate of a point in the sensing space as the local scattering rate α(Z). The scattering rate / reliability calculation unit 18 uses the degree of error that can be included in the calculation result of the local scattering rate α(Z) for each calculation in the local scattering rate α(Z) as the reliability.

[0103] For example, the scattering rate / reliability calculation unit 18 is based on the rounding error ΔL of the brightness L. round The effects on residual power P(Z) and local scattering rate α(Z) (Equations (5) and (6) below) and the deviation (standard deviation) Δα from the local scattering rate α(X±ΔX, Y±ΔY, Z±ΔZ) of surrounding pixels. SD The reliability of the estimated local scattering rate α(Z) is calculated using (Z). α (Z)(Formula (7) below).

[0104] L(Z)=P(Z-1)·(f(Z,α(Z)) / f(Z-1,α(Z)))…(3)

[0105]

[0106] Conf α (Z)=g α (Δα round (Z), Δα SD (Z))…(7)

[0107] In equation (5), ΔP round(Z) represents the rounding error ΔL of the brightness L. round Up to the integral of (Z-1). In equation (6), Δα round (Z) represents the rounding error ΔL due to the brightness L. round This causes an estimation error in the local scattering rate α(Z). In equation (7), g a (Δα round (Z), Δα SD (Z) represents the reliability scoring function. The scoring function g a Examples can include the following equation (8). However, the scoring function g a The shape is optional. For example, the scoring function g a It can be set as indicated by the following equation (9), and the reliability can be the estimation error itself.

[0108]

[0109] g α (Δα round (Z), Δα SD (Z))=Δα round (Z)+Δα SD (Z)…(9)

[0110] [2-2-2. Estimation of blurry local scattering rate based on received light image]

[0111] Figures 9 to 12 It is used to illustrate the estimation of local scattering rate (LS) based on the blurring of the received light image. P A diagram illustrating the method.

[0112] When using a blurred received light image, it is necessary to differentiate the pixel driving scheme of the LiDAR light receiving unit 36 ​​from the usual pixel driving scheme. For example, the PSF acquisition unit 16 expands the driving area DR of the sensor surface (LiDAR light receiving unit 36) driven in sync with the laser to the peripheral pixels that were not originally reached by the laser. Therefore, a wide waveform (blurred waveform) due to the blurring of the received light image is detected across multiple pixels. The PSF acquisition unit 16 extracts the blurred waveform L of points in the sensing space from the LiDAR data LD. bokeh .

[0113] Blurred waveform L bokeh This includes light-receiving data acquired from pixels corresponding to the laser scanning position and one or more surrounding pixels not corresponding to the laser scanning position. The method for selecting surrounding pixels is optional. Figure 11In the example shown, the laser scanning position is a linear region with a width of one pixel, and the surrounding pixels driven simultaneously with the laser scanning position are linear regions with a width of one or more pixels outside the laser scanning position. However, the surrounding pixels that are to be the driving region DR do not always have to be linear regions with a constant width. Figure 12 As shown in the example, the peripheral pixels to be used as the driving region DR can also be configured in a comb-like pattern on the outside of the laser scanning position.

[0114] Scattering rate / reliability calculation unit 18 is based on fuzzy waveform L bokeh To estimate the distribution of scattering rate. (Fuzzy waveform L) bokeh (Z) can be represented by the fuzzy function N(△X, α(Z)) by the following equation (10). The fuzzy function N(△X, α(Z)) is, for example, a normal distribution, the width of which is determined by the local scattering rate α(Z). The scattering rate / reliability calculation unit 18 estimates the local scattering rate α(Z) of a point in the sensing space by applying the measured value to equation (10).

[0115] L bokeh (Z)=L(Z)·N(ΔX,α(Z))…(10)

[0116] The advantage of this example is that the local scattering rate α(Z) can be estimated without relying on the waveform history of the brightness in the Z direction (residual power P). Note that in this example, it is also possible to... Figure 8 As shown in the example, the reliability of calculating the local scattering rate α(Z) is based on the rounding error of the brightness and the deviation from the surrounding pixels.

[0117] [2-2-3. Subject Depth Extraction Based on Local Scattering Rate]

[0118] Figure 13 This is a diagram illustrating an example of a method for extracting the subject depth DP based on the local scattering rate α(Z).

[0119] To improve the accuracy of the local scattering rate α(Z), the scattering rate / reliability calculation unit 18 determines the signal from a spatial region outside of a scattering medium such as rain or fog (subject SB). The scattering exclusion unit 13 estimates the subject depth DP based on the determined signal. This is because, in the presence of an object outside the scattering medium, reflected light from the object dominates, making it difficult to accurately fit the brightness data into the scattering model.

[0120] For example, the scattering rate / reliability calculation unit 18, based on the magnitude, rate of change, and reliability of the local scattering rate α(Z), identifies the interval from which the signal from the scattering medium is estimated to be dominant as the scattering-determined interval SD. The scattering exclusion unit 13 selectively uses LiDAR data from intervals outside the scattering-determined interval SD (non-scattering-determined intervals) to extract the subject depth DP.

[0121] exist Figure 13 In the example shown, if any of the following conditions are met for the first time, the scattering rate / reliability calculation unit 18 detects the region in the sensing space along the z-direction up to the location where the condition is first met as the scattering determination interval SD. The scattering exclusion unit 13 determines the subject depth DP based on the maximum peak value in the interval other than the scattering determination interval SD (non-scattering determination interval).

[0122] Condition 1: The value of the local scattering rate α(Z) is equal to or greater than the threshold.

[0123] Condition 2: The rate of increase of the local scattering rate α(Z) is equal to or greater than the threshold.

[0124] Condition 3: The reliability of the local scattering rate α(Z) is less than the threshold.

[0125] [2-3. LiDAR Output Unit]

[0126] Figure 14 and Figure 15 This is a diagram used to illustrate the processing of LiDAR output unit 14. Figure 14 This shows the data input to the LiDAR output unit 14 (local scattering rate LS). P Example diagram.

[0127] Figure 15 This shows the data output from LiDAR output unit 14 (local scattering rate LS). B Example diagram.

[0128] LiDAR output unit 14 obtains local scattering rate LS P Extract eigenvalues ​​and use the extracted eigenvalue data as the local scattering rate (LS). B The output is sent to the fusion unit 15. For example, the LiDAR output unit 14 divides the sensing space into multiple blocks. The LiDAR output unit 14 generates the local scattering rate LS of at least one block included in the scattering determination interval SD of the multiple divided blocks. P The representative value is used as correction information for correcting the IM of RGB images.

[0129] The division can be performed in units of echo EC (echo curve), or in units other than echo EC (e.g., optionally, spatial units). Echo EC signifies the period from the beginning to the end of the mountain range. In cases where multiple ridges are connected, division can be performed in the valleys. Figure 14 In the example shown, the data is divided into units called echo ECs. An echo EC is a block. For example, LiDAR output unit 14 divides the sensing space such that each of the multiple blocks includes at least one peak appearing in the waveform of the LiDAR data LD.

[0130] exist Figure 14 In the example shown, N echo ECs are generated along the Z-axis. "Echo 1" through "Echo 4" are echo groups indicating scattered echoes, and "Echo 5" through "Echo N" are echo groups indicating subject echoes. Scattered echoes are echo ECs primarily composed of scattered light from a scattering medium (rain, fog, etc.) present within the scattering defined interval SD. Subject echoes are echo ECs existing outside the scattering defined interval SD (non-scattering defined interval) and primarily composed of reflected light from an object other than the scattering medium (the subject).

[0131] LiDAR output unit 14 obtains local scattering rate LS P The waveform data (WV) and metadata (MT) of the blocks are extracted from the data. The LiDAR output unit 14 generates output data including the waveform data (WV) and metadata (MT) and outputs it to the fusion unit 15.

[0132] Waveform data WV includes waveform information WI for the blocks. Waveform information WI refers to information related to the characteristic values ​​required to reproduce the waveform. For example, waveform information WI includes information about the peak height of the waveform, the peak position of the waveform, the start position of the echo EC, the end position of the echo EC, and the half-width of the echo EC. Position refers to the position in the z-axis direction.

[0133] Metadata (MT) includes the metadata of a block. Metadata refers to additional information about the waveform itself that is associated with the waveform within the block. For example, metadata includes the scattering determination tag (LB), scattering rate information (SI), and scattering rate reliability information (CF).

[0134] The scattering determination tag (LB) is a label indicating whether an echo (EC) in a block is a scattered echo. For example, a scattered echo is assigned a "1" as the scattering determination tag LB. A subject echo is assigned a "0" as the scattering determination tag LB. Echo ECs with a scattering determination tag LB of "0" are identified as scattered echoes and are considered invalid data in the calculation of subject depth (DP). Note that subject depth (DP) refers to the waveform information WI of the echo ECs (subject echoes) with a scattering determination tag LB of "0" extracted from the waveform data.

[0135] Scattering rate information SI refers to the representative value of the local scattering rate within the echo interval (from the start to the end of the echo EC) in the block. Scattering rate reliability information CF refers to the representative value of the reliability of the local scattering rate within the echo interval in the block. Examples of representative values ​​include the average and maximum values.

[0136] Fusion unit 15 uses local scattering rate LS B The data is used to perform RGB image IM correction and sensor fusion processing. In these types of processing, the scattering rate information SI and scattering rate reliability information CF of the scattered echo group, the waveform information WI (subject depth DP) of the subject echo group, and the scattering determination label LB for identifying invalid information are required.

[0137] Included in local scattering rate LS B Other information can be used as verification information for sensor fusion, for debugging and reliability improvement. However, this type of information does not always have to be output to the fusion unit 15. The LiDAR output unit 14 can also be... Figure 15 The waveform data WV and metadata MT shown are used to extract a portion of the information required for sensor fusion, and selectively output only the extracted information to the fusion unit 15. By outputting only the necessary information, the amount of data transmission is reduced.

[0138] For example, the echo EC in the non-scattering deterministic interval is the subject echo caused by reflected light from an object outside the scattering medium. Since the scattering rate is defined with respect to the scattering medium, the scattering rate information SI of the subject echo has no physical meaning. Therefore, only the scattering rate information SI and scattering rate reliability information CF of the echo ECs ("Echo 1" to "Echo 4") in the scattering deterministic interval SD can be output, and the output of the scattering rate information SI and scattering rate reliability information CF of the echo ECs ("Echo 5" to "Echo N") in the non-scattering deterministic interval, which are unnecessary information, can be omitted.

[0139] The reduction of output items is not limited to scattering rate information SI and scattering rate reliability information CF. A portion of waveform information WI can be reduced from the output items. Figures 16 to 18 This is a diagram illustrating an example of a method for reducing waveform information (WI).

[0140] When acquiring depth information of the subject, the waveform information WI of the scattering determination interval SD where the subject does not exist is not required. The waveform information WI of the scattering determination interval SD can be used as verification information. However, it is not necessary to output the entire waveform information WI to the fusion unit 15. As verification information, only some waveform characteristics included in the scattering determination interval SD or the global waveform characteristics of the scattering determination interval SD need to be used.

[0141] exist Figure 16 In the example shown, only some waveform characteristics included in the scattering determination interval SD are output to the fusion unit 15. The LiDAR output unit 14 extracts waveform information WI of at least one block of the scattering determination interval SD from the LiDAR data LD. The LiDAR output unit 14 outputs the extracted waveform information WI as verification information for sensor fusion. The number of blocks used for verification information (e.g., the number of echo ECs) is less than the total number of blocks included in the scattering determination interval SD.

[0142] For example, LiDAR output unit 14 extracts at least one scattered echo from the scattering determination interval SD as a representative echo. LiDAR output unit 14 selectively outputs the waveform information WI of the representative echo to fusion unit 15. The method for selecting the representative echo is optional. Figure 16 In the example shown, the scattered echoes from the head of the scattered echo group ("Echo 1" and "Echo 2") were selected as representative echoes.

[0143] exist Figure 17 In the example shown, only the global waveform characteristics of the scattering-determined interval (SD) are output to the fusion unit 15. For example, the LiDAR output unit 14 smooths the waveform of the scattering-determined interval (SD) using a method such as moving average. The LiDAR output unit 14 outputs the waveform information obtained by smoothing the LiDAR data LD in the scattering-determined interval (SD) as verification information for sensor fusion. This method can be used when sensor fusion can be verified using only global information.

[0144] exist Figure 18 In the example shown, the amount of data is reduced by sharing data from adjacent pixels. Data sharing is performed on the scattering determination interval SD. Within the scattering determination interval SD, the scattering medium is dispersed such that its density changes slowly. Therefore, the waveforms of the scattered echoes are similar between adjacent pixels, and data can be shared.

[0145] [2-4. Fusion Unit]

[0146] Figure 19 This is a diagram showing the specific form of the fusion unit 15.

[0147] Prior to sensor fusion, fusion unit 15 corrects the RGB image IM, which is a combination of sensor fusion and LiDAR data LD, based on the scattering rate distribution. The fusion unit 15 then corrects the RGB image IM based on the scattering rate distribution (the corrected image IM). R The sensor performs fusion with the subject's depth DP and generates a corrected depth DP as high-precision depth information. R .

[0148] For example, the fusion unit 15 includes an RGB scattering removal unit 21, a combination ratio acquisition unit 22, an object recognition unit 23, an optical flow acquisition unit 24, a deep integration unit 25, and an object tracking unit 26. The RGB scattering removal unit 21, the combination ratio acquisition unit 22, the object recognition unit 23, the optical flow acquisition unit 24, and the deep integration unit 25 are implemented, for example, by a neural network (feature extraction network unit) for feature extraction.

[0149] RGB scattering removal unit 21 is based on local scattering rate LS B The data is used to correct the RGB image IM to generate the corrected image IM. R Corrected image IM R The generation process is as follows.

[0150] <Process 1> Based on the subject depth DP and local scattering rate LS B The initial value t0 of the estimated RGB degradation amount is calculated.

[0151] <Process 2> Correction of RGB degradation and removal of scattering components from RGB image IM.

[0152] First, as process 1, the RGB scattering removal unit 21 considers the variation of the local scattering rate α in the z-direction and calculates the estimated initial value t0 of the RGB degradation amount t based on the following equation (11). In equation (11), Z obj This represents the depth in the z-direction extracted from the subject's depth DP.

[0153]

[0154] Next, as process 2, the RGB scattering removal unit 21 estimates the RGB components of ambient light scattering (atmospheric scattered light A). Then, the RGB scattering removal unit 21 corrects the RGB degradation amount t according to the estimated initial value t0 and removes the scattering components from the RGB image I based on the following equation (12).

[0155] I(X,Y,C)=t(X,Y)J(X,Y,C)+(1-t(X,Y))A…(12)

[0156] <Estimation of Atmospheric Scattered Light>

[0157] Atmospheric scattered light A is determined based on pixel data of a pixel group that has sufficient distance from the subject and mainly consists of scattered components. For example, the RGB scattering removal unit 21 determines atmospheric scattered light A as satisfying Zobj(X,Y)≥Z obj_thresh The average RGB value in the pixel group. System developers can optionally set the threshold Z. obj_thresh.

[0158] <Estimation of RGB degradation t1 based on RGB image I>

[0159] For example, the RGB scattering removal unit 21 determines the RGB degradation amount t1 based on the method of Non-Patent Document 1 described later, according to the RGB image I and atmospheric scattered light A.

[0160] Non-patent literature 1: He, Kaiming et al., “Single Image Haze Removal Using DarkChannelPrior”. Conference on Computer Vision and Pattern Recognition, 2009.

[0161] For the aforementioned alternative, the RGB degradation amount t1 can also be determined through network-based learning and inference. As a network-based learning and inference method, the method described in Non-Patent Document 2 below can be used. From Non-Patent Document 2... Figure 2 The process of obtaining the estimated transmittance from the blurred image (extracted via CNN and random forest regression) corresponds to the learning and inference methods described above. The RGB degradation t1 can be obtained from the RGB image I using the same method.

[0162] Non-patent literature 2: Song, Yafei et al., “Single Image Dehazing Using Ranking Convolutional Neural Network”, arXiv, 2020.

[0163] <Correction of Scattering Rate>

[0164] The RGB scattering removal unit 21 mixes the estimated initial value t0 and the RGB degradation amount t1 to determine the final RGB degradation amount t(X, Y). For example, the ratio parameter of the estimated initial value t0 is represented as c0, and the ratio parameter of the RGB degradation amount t1 is represented as c1 (c0+c1=1). The RGB degradation amount t(X, Y) is calculated based on t(X, Y)=c0*t0(X, Y)+c1*t1(X, Y).

[0165] System developers can optionally set the ratio parameters c0 and c1. To set the ratio parameters c0 and c1, a method based on I(X, Y, C) and Z can be used. obj The prior learning model F of (X, Y). “C” represents the three channels R (red), G (green), and B (blue). The RGB scattering removal unit 21 can be based on c0 = F(I(X, Y, C), Zobj (X, Y)) to determine the ratio parameter c0.

[0166] The RGB scattering removal unit 21 can correct the local scattering rate α according to the correction amount of the RGB degradation amount. For example, the RGB scattering removal unit 21 can obtain the corrected local scattering rate α0 corrected based on α0(X, Y, Z) = α(X, Y, Z) × {t0(X, Y) / t(X, Y)} as the corrected scattering rate α0.

[0167] <RGB scattering exclusion>

[0168] The RGB scattering removal unit 21 obtains the corrected image J according to Equation (12) based on the RGB scattering model.

[0169] Note that the above processes 1 and 2 can be end-to-end processes that perform network-based learning and inference. For example, the network (end-to-end CNN) shown in Non-Patent Document 3 described later can be used for this processing. Figure 1 shown network (end-to-end CNN) for this processing.

[0170] Non-Patent Document 3: Li, Jinjiang, et al., "Image Dehazing Using Residual-Based DeepCNN", IEEE Access, 2018, Vol. 6.

[0171] In the case of using the end-to-end CNN of Non-Patent Document 3, in addition to the RGB image I(X, Y, C), the depth Z obj (X, Y) and the local scattering rate α(X, Y, Z) need to be added to the input. As an extended method of the present disclosure, for example, in the input eigenvalue map I_feature(X, Y, C) of the network, the depth Z obj (X, Y) and the local scattering rate α(X, Y, Z) only need to be accumulated after the RGB three layers (C = 0, 1, and 2).

[0172] When the RGB image I(X, Y, C), the depth Z obj (X, Y) and the local scattering rate α(X, Y, Z) are input to the network as the input eigenvalue map, the depth Z obj (X, Y) and the corrected scattering rate α0(X, Y, Z) are output as the output eigenvalue map.

[0173] The combination ratio acquisition unit 22 determines the combination ratio RA in sensor fusion based on the subject depth DP and the corrected image IM R to determine the combination ratio RA. The combination ratio RA is determined based on the reliability of each sensor data (LiDAR data LD and RGB image IM) to be combined.

[0174] For example, the combination ratio acquisition unit 22 calculates the reliability (LiDAR reliability) of the LiDAR data LD at a point in the sensing space based on the signal strength of the subject depth DP and the amount of degradation of the RGB image IM caused by scattering. LiDAR Reliability Conf LiDAR The calculation of (X, Y) is based on the following equation (13). In equation (13), scale dist and scale peak This refers to the scaling factor (scaling factor) used to scale the depth and peak values.

[0175]

[0176] The combination ratio acquisition unit 22 calculates the corrected RGB image IM based on the comparison result between the color of the RGB image IM and the color of the scattered light. R The reliability of the scattered light (RGB reliability). The color of the scattered light refers to the typical color emitted by the scattered light, such as white or gray. System developers can optionally set the color of the scattered light. RGB Reliability Conf RGB (X, Y) is calculated based on the following equation (14).

[0177]

[0178] In equation (14), △p i It refers to the feature point p near (X, Y). i The amount of movement A (parallax) between stereoscopic images or frames. point This refers to the scaling factor used to scale the amount of movement. (in the context of scale) color The term in the denominator indicates the difference between the color of the scattered light and the color it appears as (white, gray, etc.). color This refers to the scaling factor used to scale color differences.

[0179] Combination ratio acquisition unit 22 Point-based LiDAR reliability Conf LiDAR (X, Y) and RGB reliability Conf RGB (X, Y) is used to determine the combination ratio RA of points in sensor fusion. The combination ratio C of RGB image IM is... RGB (X, Y) and C of LiDAR data LD LiDAR The calculation of (X, Y) is based on the following equation (15).

[0180]

[0181] In the example above, the reliability and combination ratio RA of the sensor data are determined based on the subject's feature values ​​(depth, color, etc.) obtained from individual sensor data (LiDAR data LD and RGB image IM). However, the feature values ​​used to determine reliability and the formulas for calculating reliability are not limited to those described above. The feature values ​​and formulas used to determine reliability only need to be those used to evaluate the discriminative power of individual sensor data.

[0182] The object recognition unit 23 is based on the subject depth DP and the corrected image IM. R To identify objects. Object recognition unit 23 assigns an object recognition label OR to the identified objects. Optical flow acquisition unit 24 acquires the optical flow based on the subject depth DP and the corrected image IM. R To acquire optical flow (OF) (subject motion, MO). The depth integration unit 25 combines the subject depth (DP) and the corrected image depth (IM) based on the aforementioned combination ratio (RA). R Therefore, LiDAR data (LD) and RGB image (IM) are integrated to generate a corrected depth DP as high-precision depth information. R .

[0183] Object tracking unit 26 is based on object identification tag (OR), optical flow (OF), and calibrated depth (DP). R To track objects. The object tracking unit 26 assigns an identification tag RL with tracking information to the tracked object.

[0184] [3. Revision]

[0185] [3-1. Feedback of corrected scattering rate]

[0186] Figure 20 This shows the local scattering rate LS P The modified graph shows the estimation process. In this modification, the corrected scattering rate LS calculated by fusion unit 15 is used. R Feedback is sent back to the scattering rate estimation unit 12 to improve the local scattering rate LS P The processing of estimation accuracy.

[0187] For example, the corrected scattering rate fed back from the fusion unit 15 to the scattering rate estimation unit 12 is represented as α0, and the local scattering rate re-estimated by the scattering rate estimation unit 12 is represented as α. The scattering rate / reliability calculation unit 18 mixes the corrected scattering rate α0 and the local scattering rate α based on the following equation (16). Therefore, a high-precision local scattering rate α' is obtained. In equation (16), C brend This indicates the mixing ratio. System developers can optionally set the mixing ratio C. brend .

[0188] α'=Cbrend α0(Z)+(1-C brend )α(Z)…(16)

[0189] The scattering rate / reliability calculation unit 18 obtains the difference Δα between the corrected scattering rate α0 and the local scattering rate α based on the following equation (17). diff (Z). The scattering rate / reliability calculation unit 18 calculates the difference Δα based on the following equation (18). diff (Z) Add to the reliability scoring function g a Therefore, the accuracy of reliability is improved.

[0190] Δα diff (Z)=|α(Z)-α0(Z)|…(17)

[0191]

[0192] [3-2. Acquisition of multiple LiDAR data with different laser powers]

[0193] Figure 21 and Figure 22 This is a diagram illustrating examples of acquiring multiple LiDAR data LDs with different laser powers.

[0194] In this modification, both high-power and low-power lasers are used to acquire LiDAR data (LD). Measurements using high-power and low-power lasers each have their advantages and disadvantages. For example, measurements using high-power lasers can accurately measure distant objects, but the signal of nearby scattered light tends to saturate. Measurements using low-power lasers can accurately measure the signal of nearby scattered light, but it is difficult to accurately measure distant objects. This modification proposes a method to perform wide-range measurements with high accuracy by compensating for the shortcomings of the respective measurements.

[0195] For example, the scattering rate / reliability calculation unit 18 estimates the local scattering rate LS of the near-field based on LiDAR data LD measured by a low-power laser. P The scattering rate / reliability calculation unit 18 estimates the local scattering rate LS of the distant view based on LiDAR data LD measured by a high-power laser. P The scattering exclusion unit 13 estimates the subject depth DP in the distant view based on LiDAR data LD measured by a high-power laser.

[0196] The near field refers to the range where the brightness of a LiDAR data LD using a high-power laser exceeds the permissible standard. The range outside the defined near field is the far field. System developers can optionally set the permissible standards. Figure 22 In the example shown, the area where the brightness exceeds the threshold is defined as the foreground.

[0197] "High power" and "low power" are relative terms and do not define numerical ranges. In this revision, the laser with the relatively higher power among two lasers with different power is described as a "high-power laser," and the laser with the relatively lower power is described as a "low-power laser."

[0198] [3-3. Changes in data format]

[0199] Figure 23 This is a graph showing the changes in the output data of the LiDAR output unit 14.

[0200] The LiDAR output unit 14 divides the scattering determination interval SD into multiple blocks, and acquires scattering rate information SI and scattering rate reliability information CF for each block. Figure 14 In the example shown, information is acquired in units of echo EC. However, in this modification, information is acquired from individual intervals SE, which are set independently of the echo EC.

[0201] exist Figure 23 In the example shown, each interval SE, divided by a constant width, is set as a block, and scattering rate information SI and scattering rate reliability information CF are obtained from these blocks ("Interval 1" to "Interval M"). Since the scattering determination tag LB is a characteristic value of the accompanying echo EC, the scattering determination tag LB is set in units of echo EC.

[0202] Scattering rate information (SI) and scattering rate reliability information (CF) differ from waveform information (WI) and scattering determination tag (LB). In scattering rate information (SI) and scattering rate reliability information (CF), the unit of information acquisition is each interval (SE), and information is acquired in units of echo (EC). Therefore, scattering rate information (SI) and scattering rate reliability information (CF) can be transmitted separately from waveform information (WI) and scattering determination tag (LB).

[0203] [3-4. Feedback on corrected depth]

[0204] Figure 24 This diagram illustrates the modifications made to the IM (Intensity Dispersion) processing for removing scattering and the acquisition of the combination ratio RA (Range Reduction) from an RGB image. In this modification, the corrected depth DP (Depth Percentage) acquired in the immediate preceding frame is used. R Feedback is sent back to the RGB scattering removal unit 21 and the combination ratio acquisition unit 22 to improve the corrected image IM. R And the processing of the accuracy of the combination ratio RA.

[0205] [3-5. Re-rendering of the scattered RGB image]

[0206] Figure 25 This is a diagram showing a re-rendered RGB image of the scattered image. The following explanation will further elaborate on this. Figure 19 The differences in the configurations shown.

[0207] The fusion unit 15 includes a point cloud image generation unit 27 and a scattering regeneration unit 28. The point cloud image generation unit 27 is based on the corrected image IM. R Corrected depth DP R Corrected scattering rate LS R Point cloud map (PC) is generated using atmospheric scattered light (SC). The PC includes information related to the distribution of scattering particles and the distribution of scattering rate in the sensing space.

[0208] The scattering regeneration unit 28 renders a point cloud map PC from an arbitrary viewpoint and generates a rendered image IM that includes the scattering of scattering particles. C Rendered image IM C It is generated as an RGB image (scattered RGB image) that reproduces the color of atmospheric scattered light, the concentration of scattering particles (such as fog), etc.

[0209] For the generation of the point cloud image PC, the scattering model indicated by equation (12) can be used. This is used to recover the corrected image IM from the RGB image IM. R In the processing, the RGB degradation t and atmospheric scattered light A are estimated. This can be based on the corrected depth DP. R The data is used to generate a point cloud of the subject. The point cloud can then be used to generate an RGB image of the subject viewed from any viewpoint.

[0210] The RGB degradation t observed from any viewpoint can be recalculated based on the depth after viewpoint transformation and the scattering rate of scattering particles in the light path. By applying the recalculated RGB degradation t together with the RGB image J and atmospheric scattered light A to Equation (12), an RGB image I (rendered image IM) including scattering, observed from any viewpoint, can be regenerated. C ).

[0211] [4. Hardware Configuration Example]

[0212] Figure 26 This is a diagram illustrating an example of the hardware structure of the fusion system 10.

[0213] Information processing in the fusion system 10 is performed, for example, by a computer 1000. The computer 1000 includes a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read-Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The units of the computer 1000 are connected via a bus 1050.

[0214] The CPU 1100 operates and controls the unit based on the program (program data 1450) stored in the ROM 1300 or HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes the processing corresponding to the various programs.

[0215] ROM 1300 stores boot programs such as the BIOS (Basic Input / Output System) and programs that depend on the hardware of computer 1000, which are executed by CPU 1100 when computer 1000 starts up.

[0216] HDD 1400 is a non-transitory computer-readable recording medium that non-transitoryly records programs to be executed by CPU 1100, data to be used by the program, etc. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment, which is an example of program data 1450.

[0217] Communication interface 1500 is an interface for computer 1000 to connect to external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from other devices or sends data generated by CPU 1100 to other devices via communication interface 1500.

[0218] Input / output interface 1600 is an interface used to connect input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as keyboards or mice via input / output interface 1600. CPU 1100 sends data to output devices such as display devices, speakers, or printers via input / output interface 1600. Input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium. The medium may be, for example, optical recording media such as DVDs (Digital Multifunction Optical Discs) or PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, semiconductor memory, etc.

[0219] For example, when computer 1000 is used as the fusion system 10 according to the embodiment, the CPU 1100 of computer 1000 performs the functions of the above-described unit by executing an information processing program loaded on RAM 1200. HDD 1400 stores information processing programs, various models, and various data according to this disclosure. Note that CPU 1100 reads program data 1450 from HDD 1400 and executes program data 1450. However, as another example, CPU 1100 may obtain these programs from other devices via external network 1550.

[0220] [5. Effects]

[0221] The fusion system 10 includes a scattering rate estimation unit 12 and a fusion unit 15. The scattering rate estimation unit 12 estimates the distribution of scattering rate in the sensing space based on LiDAR data LD. Before sensor fusion, the fusion unit 15 corrects the sensor data combined with the LiDAR data LD through sensor fusion based on the scattering rate distribution. In the information processing method of this disclosure, the processing of the fusion system 10 is performed by a computer 1000. The computer-readable non-transitory storage medium of this disclosure stores a program for enabling the computer 1000 to implement the processing of the fusion system 10.

[0222] This configuration allows for the appropriate elimination of scattering effects from the output information obtained through sensor fusion. Therefore, a fusion system 10 with high resilience to severe weather conditions is constructed.

[0223] The scattering rate estimation unit 12 estimates the distribution of scattering rate based on light reception data obtained from pixels corresponding to the laser scanning position and peripheral pixels not corresponding to the laser scanning position.

[0224] Using this configuration, the driving region DR of the sensor surface, driven synchronously with the laser, is expanded to include peripheral pixels that were not originally reached by the laser beam. The blurring of the received light image caused by scattering is detected as a wide waveform spanning multiple pixels, thus improving the accuracy of scattering rate estimation.

[0225] The fusion system 10 includes a scattering exclusion unit 13. The scattering exclusion unit 13 excludes components caused by scattering from the LiDAR data LD based on the distribution of scattering rate, and extracts the waveform information WI, which excludes the influence of scattering, as the subject depth DP.

[0226] Using this configuration, it is possible to recover the subject depth DP before scattering occurs.

[0227] The fusion unit 15 performs sensor fusion on sensor data corrected based on the scattering rate distribution and the subject depth DP, and generates high-precision depth information.

[0228] Using this configuration, depth information with reduced scattering effects can be obtained.

[0229] The fusion unit 15 calculates the reliability of the LiDAR data LD at a point in the sensing space based on the signal strength of the subject depth DP and the amount of sensor data degradation caused by scattering. The fusion unit 15 then determines the combination ratio RA of the points in the sensor fusion based on the reliability of the LiDAR data LD at the point.

[0230] This configuration allows for appropriate combinations based on the reliability of the data.

[0231] The fusion unit 15 calculates the reliability of points in the corrected RGB image IM based on a comparison between the colors of the RGB image IM acquired as sensor data and the colors of the scattered light. The fusion unit 15 then determines the combination ratio RA of points in the sensor fusion based on the reliability of the points.

[0232] Using this configuration, the scattering intensity at each location is estimated based on the concentration of the color rendered by the scattered light. By adjusting the combination ratio at each location based on the scattering intensity, highly reliable output information can be obtained.

[0233] The scattering rate estimation unit 12 includes a scattering rate / reliability calculation unit 18. The scattering rate / reliability calculation unit 18 acquires the scattering rate of a point in the sensing space as the local scattering rate LS. P Scattering rate / reliability calculation unit 18 is for local scattering rate LS P Each calculation in the process can be included in the local scattering rate LS P The degree of error in the calculation results is used as the reliability.

[0234] This configuration allows for the generation of highly reliable output information.

[0235] Scattering Rate / Reliability Calculation Unit 18 is based on Local Scattering Rate LS P The magnitude, rate of change, and reliability of the signal from the scattering medium are estimated to determine the dominant interval detection as the scattering-determined interval (SD). The scattering exclusion unit 13 selectively uses LiDAR data LD from intervals outside the scattering-determined interval (SD) to extract the subject depth (DP).

[0236] This configuration prevents signals generated by changes in the concentration of scattering particles from being mistakenly detected as signals from the subject.

[0237] The fusion system 10 includes a LiDAR output unit 14. The LiDAR output unit 14 divides the sensing space into multiple blocks. The LiDAR output unit 14 generates the local scattering rate LS of at least one block included in the scattering determination interval SD of the multiple divided blocks. P The representative value is used as correction information to correct sensor data.

[0238] Using this configuration, the local scattering rate LS is compressed on a block-by-block basis. P Data based on local scattering rate LS. P The compressed data is used to properly correct the sensor data.

[0239] The LiDAR output unit 14 divides the sensing space such that each of the multiple blocks includes at least one peak appearing in the waveform of the LiDAR data.

[0240] Using this configuration, the local scattering rate LS can be extracted appropriately. P While improving the waveform characteristics, it also increases the data compression rate.

[0241] The LiDAR output unit 14 outputs waveform information WI of at least one block of the scattering determination interval SD extracted from the LiDAR data LD as verification information for sensor fusion.

[0242] This configuration allows for the effective use of information related to the scattering interval, which is otherwise unrelated to the measurement of the subject, thereby improving the accuracy of sensor fusion.

[0243] The number of blocks used for verification information is less than the total number of blocks included in the scattering determination interval SD.

[0244] This configuration allows for the verification of sensor fusion with less information.

[0245] The LiDAR output unit 14 outputs the waveform information obtained by smoothing the LiDAR data LD in the scattering determination interval SD as the verification information of sensor fusion.

[0246] Using this configuration, the characteristics of the global waveform of the SD region can be appropriately verified based on the scattering.

[0247] Scattering rate estimation unit 12 estimates the local scattering rate LS of the near scene based on LiDAR data LD measured by a low-power laser. P The scattering rate estimation unit 12 estimates the local scattering rate LS of the distant scene based on LiDAR data LD measured by a high-power laser. P The scattering exclusion unit 13 estimates the subject depth DP in the distant view based on LiDAR data LD measured by a high-power laser.

[0248] This configuration enables high-precision measurements over a wide range while compensating for the drawbacks of using both high-power and low-power lasers.

[0249] The fusion unit 15 includes a point cloud image generation unit 27 and a scattering regeneration unit 28. The point cloud image generation unit 27 generates a point cloud image PC that includes information related to the distribution of scattering rate in the sensing space. The scattering regeneration unit 28 renders the point cloud image PC from any viewpoint and generates a rendered image IM including scattering. C .

[0250] Using this configuration, for example, the hazy state caused by scattering can be accurately reproduced from any viewpoint.

[0251] Note that the effects described in this instruction manual are illustrative only and not limiting. Other effects may exist.

[0252] [Additional Explanation]

[0253] Note that this technology can also be configured as follows. (1)

[0255] An information processing device, comprising:

[0256] A scattering rate estimation unit that estimates the distribution of scattering rate in the sensing space based on LiDAR data; and

[0257] The fusion unit corrects the sensor data to be combined with the LiDAR data through the sensor fusion based on the distribution of the scattering rate before sensor fusion. (2)

[0259] According to the information processing apparatus described in (1), wherein,

[0260] The scattering rate estimation unit estimates the distribution of the scattering rate based on light reception data obtained from pixels corresponding to the scanning position of the laser and surrounding pixels that do not correspond to the scanning position of the laser. (3)

[0262] The information processing apparatus according to (1) or (2) includes a scattering exclusion unit, which excludes components caused by scattering from the LiDAR data based on the distribution of the scattering rate, and extracts waveform information after scattering has been excluded as the subject depth. (4)

[0264] According to the information processing device described in (3), wherein,

[0265] The fusion unit performs sensor fusion on the sensor data corrected based on the scattering rate distribution and the subject depth, and generates high-precision depth information. (5)

[0267] According to the information processing apparatus described in (4), wherein,

[0268] The fusion unit calculates the reliability of LiDAR data at each point in the sensing space based on the signal strength at the depth of the subject and the amount of degradation of the sensor data due to scattering, and determines the combination ratio of the points in the sensor fusion based on the reliability of the LiDAR data at each point. (6)

[0270] According to the information processing apparatus described in (5), wherein,

[0271] The fusion unit calculates the reliability of each point in the corrected RGB image based on a comparison between the color of the RGB image acquired as sensor data and the color of the scattered light, and determines the combination ratio of each point in the sensor fusion based on the reliability of each point. (7)

[0273] The information processing apparatus according to any one of (3) to (6), wherein,

[0274] The scattering rate estimation unit includes a scattering rate / reliability calculation unit, which obtains the scattering rate of each point in the sensing space as a local scattering rate, and calculates the degree of error that can be included in the calculation result of the local scattering rate for each of the local scattering rates as the reliability. (8)

[0276] According to the information processing apparatus described in (7), wherein,

[0277] The scattering rate / reliability calculation unit, based on the magnitude, rate of change, and reliability of the local scattering rate, detects the interval where the signal from the scattering medium is estimated to be dominant as the scattering determination interval, and...

[0278] The scattering exclusion unit selectively uses LiDAR data from intervals other than the scattering determination interval to extract the subject depth. (9)

[0280] The information processing apparatus according to (8) includes a LiDAR output unit, which divides the sensing space into multiple blocks and generates representative values ​​of the local scattering rates of at least one block in the scattering determination interval as correction information for correcting the sensor data. (10)

[0282] According to the information processing apparatus described in (9), wherein,

[0283] The LiDAR output unit divides the sensing space such that each of the plurality of blocks includes at least one peak appearing in the waveform of the LiDAR data. (11)

[0285] According to the information processing apparatus described in (9) or (10), wherein,

[0286] The LiDAR output unit outputs waveform information of at least one block of the scattering determination interval extracted from the LiDAR data as verification information for the sensor fusion. (12)

[0288] According to the information processing apparatus described in (11), wherein,

[0289] The number of blocks used for verification information is less than the total number of blocks included in the scattering determination interval. (13)

[0291] According to the information processing apparatus described in (9) or (10), wherein,

[0292] The LiDAR output unit outputs waveform information obtained by smoothing the LiDAR data in the scattering determination range as verification information for sensor fusion. (14)

[0294] The information processing apparatus according to any one of (7) to (13), wherein,

[0295] The scattering rate estimation unit estimates the local scattering rate of the near-field based on the LiDAR data measured using low-power lasers.

[0296] The scattering rate estimation unit estimates the local scattering rate of the distant scene based on the LiDAR data measured using a high-power laser, and

[0297] The scattering exclusion unit estimates the depth of the subject in the distant view based on the LiDAR data measured using the high-power laser. (15)

[0299] The information processing apparatus according to any one of (1) to (14), wherein,

[0300] The fusion unit includes: a point cloud generation unit that generates a point cloud map including information about the distribution of the scattering rate in the sensing space; and a scattering regeneration unit that renders the point cloud map from any viewpoint and generates a rendered image including scattering. (16)

[0302] An information processing method executed by a computer, the information processing method comprising:

[0303] Estimating the distribution of scattering rate in the sensing space based on LiDAR data; and

[0304] Prior to sensor fusion, the sensor data to be combined with the LiDAR data through sensor fusion is corrected based on the distribution of scattering rate. (17)

[0306] A computer-readable non-transitory storage medium storing a program that enables a computer to perform:

[0307] Estimating the distribution of scattering rate in the sensing space based on LiDAR data; and

[0308] Prior to sensor fusion, the sensor data combined with the LiDAR data through sensor fusion is corrected based on the distribution of scattering rate.

[0309] List of reference numerals

[0310] 10. Fusion System (Information Processing Device)

[0311] 12 Scattering Rate Estimation Units

[0312] 13 Scattering Exclusion Units

[0313] 14 LiDAR output units

[0314] 15 fusion units

[0315] 18 Scattering Rate / Reliability Calculation Unit

[0316] 27-point cloud map generation unit

[0317] 28 scattering regeneration units

[0318] DP Subject Depth

[0319] DP R Corrected depth (high-precision depth information)

[0320] IM RGB image

[0321] IM C Rendered image

[0322] LD LiDAR data

[0323] LS P Local scattering rate

[0324] PC point cloud map

[0325] RA combination ratio

[0326] SD scattering determines the range

[0327] WI waveform information

Claims

1. An information processing apparatus, comprising: The scattering rate estimation unit estimates the distribution of scattering rate in the sensing space based on LiDAR data; as well as The fusion unit corrects the sensor data to be combined with the LiDAR data through the sensor fusion based on the distribution of the scattering rate before sensor fusion.

2. The information processing apparatus according to claim 1, wherein, The scattering rate estimation unit estimates the distribution of the scattering rate based on light reception data obtained from pixels corresponding to the scanning position of the laser and surrounding pixels that do not correspond to the scanning position of the laser.

3. The information processing apparatus according to claim 1, comprising a scattering exclusion unit, wherein the scattering exclusion unit excludes components caused by scattering from the LiDAR data based on the distribution of the scattering rate, and extracts waveform information after scattering has been excluded as the subject depth.

4. The information processing apparatus according to claim 3, wherein, The fusion unit performs sensor fusion on the sensor data corrected based on the scattering rate distribution and the subject depth, and generates high-precision depth information.

5. The information processing apparatus according to claim 4, wherein, The fusion unit calculates the reliability of LiDAR data at each point in the sensing space based on the signal strength at the depth of the subject and the amount of degradation of the sensor data due to scattering, and determines the combination ratio of the points in the sensor fusion based on the reliability of the LiDAR data at each point.

6. The information processing apparatus according to claim 5, wherein, The fusion unit calculates the reliability of each point in the corrected RGB image based on a comparison between the color of the RGB image acquired as sensor data and the color of the scattered light, and determines the combination ratio of each point in the sensor fusion based on the reliability of each point.

7. The information processing apparatus according to claim 3, wherein, The scattering rate estimation unit includes a scattering rate / reliability calculation unit, which obtains the scattering rate of each point in the sensing space as a local scattering rate, and calculates the degree of error that can be included in the calculation result of the local scattering rate for each of the local scattering rates as the reliability.

8. The information processing apparatus according to claim 7, wherein, The scattering rate / reliability calculation unit, based on the magnitude, rate of change, and reliability of the local scattering rate, detects the interval where the signal from the scattering medium is estimated to be dominant as the scattering determination interval, and... The scattering exclusion unit selectively uses LiDAR data from intervals other than the scattering determination interval to extract the subject depth.

9. The information processing apparatus of claim 8, comprising a LiDAR output unit, wherein the LiDAR output unit divides the sensing space into a plurality of blocks, and generates representative values ​​of the local scattering rates of at least one of the divided blocks, including at least one block in the scattering determination interval, as correction information for correcting the sensor data.

10. The information processing apparatus according to claim 9, wherein, The LiDAR output unit divides the sensing space such that each of the plurality of blocks includes at least one peak appearing in the waveform of the LiDAR data.

11. The information processing apparatus according to claim 9, wherein, The LiDAR output unit outputs waveform information of at least one block of the scattering determination interval extracted from the LiDAR data as verification information for the sensor fusion.

12. The information processing apparatus according to claim 11, wherein, The number of blocks used for the verification information is less than the total number of blocks included in the scattering determination interval.

13. The information processing apparatus according to claim 9, wherein, The LiDAR output unit outputs waveform information obtained by smoothing the LiDAR data in the scattering determination range as verification information for sensor fusion.

14. The information processing apparatus according to claim 7, wherein, The scattering rate estimation unit estimates the local scattering rate of the near-field based on the LiDAR data measured using low-power lasers. The scattering rate estimation unit estimates the local scattering rate of the distant scene based on the LiDAR data measured using a high-power laser, and The scattering exclusion unit estimates the depth of the subject in the distant view based on the LiDAR data measured using the high-power laser.

15. The information processing apparatus according to claim 1, wherein, The fusion unit includes: a point cloud generation unit that generates a point cloud map including information about the distribution of the scattering rate in the sensing space; and a scattering regeneration unit that renders the point cloud map from any viewpoint and generates a rendered image including scattering.

16. An information processing method executed by a computer, the information processing method comprising: The distribution of scattering rate in the sensing space is estimated based on LiDAR data; as well as Prior to sensor fusion, the sensor data to be combined with the LiDAR data through sensor fusion is corrected based on the distribution of the scattering rate.

17. A computer-readable non-transitory storage medium storing a program that causes a computer to: Estimating the distribution of scattering rate in the sensing space based on LiDAR data; and Prior to sensor fusion, the sensor data to be combined with the LiDAR data through sensor fusion is corrected based on the distribution of the scattering rate.

Images