Electronic device and method

Abstract

An electronic device includes circuitry configured to detect a non-line-of-sight object based on a comparison of ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from model-based light reflection prediction.

Description

Technical Field

[0001] The present disclosure generally relates to the field of imaging, and more particularly to apparatuses and methods for multi-modal image capture. Background Art

[0002] Autonomous or semi-autonomous vehicles are equipped with environmental sensors for detecting the vehicle's surroundings, and the data from the environmental sensors is evaluated in a control unit by suitable software. Conventional 2D cameras are supplemented with other camera technologies such as stereo cameras, IR cameras, RADAR, LiDAR, and time-of-flight (ToF) cameras. Based on the information obtained by these sensors, the control unit can automatically trigger and execute braking, speed, distance, compensation, and / or avoidance action controls via appropriate actuators. The control unit can also warn or inform the vehicle driver of the distance to objects around the vehicle, and the control unit can provide assistance during parking or lane changes.

[0003] Environmental sensors typically provide their measurements in the form of point clouds. The point clouds provided by the sensors are used to obtain reliable information about possible objects in the vehicle's path or during a collision with the vehicle. In recent years, these technologies have become increasingly important as driver assistance systems and autonomous driving systems rely on technologies that enable 3D space recognition.

[0004] Although there are technologies for driver assistance, there is a general desire to improve these existing technologies. Summary of the Invention

[0005] According to a first aspect, the present disclosure provides an electronic device including circuitry configured to detect non-line-of-sight objects based on a comparison of ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection.

[0006] According to another aspect, the present disclosure provides a method including: detecting non-line-of-sight objects based on a comparison of ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection.

[0007] According to another aspect, the present disclosure provides a computer program including instructions configured to, when executed on a processor, perform detecting non-line-of-sight objects based on a comparison of ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection.

[0008] Additional aspects are set forth in the dependent claims, the following description, and the drawings. Brief Description of the Drawings

[0009] The embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0010] Figure 1 A vehicle having a driver assistance system for parking space detection and a ToF camera is schematically shown;

[0011] Figure 2 An example of the configuration of a driver assistance system for parking space detection is schematically shown;

[0012] Figure 3 An example of a point cloud generated by the ToF data path of a driver assistance system for parking space detection in a parking lot is schematically shown;

[0013] Figure 4 An example of a 3D model of scene details generated by 3D reconstruction is shown;

[0014] Figure 5 The line of sight of a driver assistance system for parking space detection is schematically shown;

[0015] Figure 6 A vehicle having a driver assistance system for parking space detection based on measurements of multipath reflections when the parking space is empty is schematically shown;

[0016] Figure 7a A first example of a vehicle model of a parked vehicle derived by a driver assistance system for parking space detection from a reconstructed 3D model is schematically shown;

[0017] Figure 7b A second example of a vehicle model of a parked vehicle derived by a driver assistance system for parking space detection from a reconstructed 3D model is schematically shown;

[0018] Figure 8 A process of determining the travel time of an illumination beam reflected between two vehicles by ray tracing based on a vehicle model as an object in a scene is schematically shown;

[0019] Figure 9a An example of a predicted photon arrival time and a measured dToF histogram when the parking space is empty is schematically shown;

[0020] Figure 9b An example of a predicted phasor and a measured iToF phasor diagram when the parking space is empty is schematically shown;

[0021] Figure 10 A vehicle having a driver assistance system for parking space detection and a ToF camera when the parking space is not empty is schematically shown;

[0022] Figure 11 A vehicle with a driver assistance system for parking space detection based on measurements of multipath reflections when the parking space is not empty is schematically shown;

[0023] Figure 12a An example of the predicted photon arrival time and the measured dToF histogram in the case where the parking space is not empty is schematically shown;

[0024] Figure 12b An example of the predicted phasor and the measured iToF phasor diagram in the case where the parking space is not empty is schematically shown;

[0025] Figure 13a A flowchart of a process for determining the status of a parking space in the case of using a dToF camera is schematically shown;

[0026] Figure 13b A flowchart of a process for determining the status of a parking space in the case of using an iToF camera is schematically shown;

[0027] Figure 14a The operating principle of a SPAD photodiode is schematically shown;

[0028] Figure 14b The basic operating principle of an indirect time-of-flight imaging system that can be used for depth sensing is schematically shown;

[0029] Figure 15 An example of a process for determining pixel values according to the "photon counting" method is schematically provided;

[0030] Figure 16 An example of implementing 3D reconstruction is shown;

[0031] Figure 17 A block diagram of an example depicting the schematic configuration of a vehicle control system;

[0032] Figure 18 A diagram that helps to explain an example of the installation positions of an out-of-vehicle information detection unit and an imaging unit;

[0033] Figure 19 A diagram of a binary Bayesian hypothesis test for classifying measurement results in the case of iToF measurement is schematically shown. DETAILED DESCRIPTION

[0034] Before Figures 1 to 19 detailedly describing the embodiments, a general explanation is made.

[0035] Embodiments described in more detail below disclose an electronic device including a circuit configured to detect non-line-of-sight objects based on a comparison between ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from model-based light reflection prediction.

[0036] The circuit may include a processor, a memory (RAM, ROM, etc.), a storage device, an input device (mouse, keyboard, camera, etc.), an output device (display (e.g., liquid crystal, (organic) light emitting diode, etc.), speaker, etc.), a (wireless) interface, etc., as is known for electronic devices (computer, smartphone, vehicle control system, etc.). In addition, it may include sensors for sensing still image or video image data (image sensor, camera sensor, video sensor, ToF sensor, SPAD sensor, LiDAR sensor, etc.), sensors for sensing fingerprints, sensors for sensing environmental parameters (e.g., radar, humidity, light, temperature), etc.

[0037] A non-line-of-sight object may, for example, be an object that is not directly visible in an image. For example, a non-line-of-sight object is not visible in the main light ray. A non-line-of-sight object may be visible, for example, in a reflection.

[0038] The circuit may be configured to obtain ToF information from a photon histogram captured by a ToF imaging sensor. For example, peaks in the photon histogram related to primary reflection, secondary reflection, and subsequent reflections may be identified.

[0039] For example, the bins of the photon histogram may represent the time it takes for photons to travel from an illumination source to a scene and back to the ToF camera. Since the speed of light is constant in a vacuum, the propagation time also represents the propagation of light.

[0040] The circuit may be configured to detect at least one of the presence, position, and speed of a non-line-of-sight object.

[0041] The circuit may be configured to determine the state of a point based on the detection of a non-line-of-sight object.

[0042] The state of a point may, for example, include information about whether the point is empty.

[0043] A point may, for example, be a parking space, and the state of the point may, for example, include information about whether the parking space is empty.

[0044] In some embodiments, the ToF information includes information about the arrival time of photons related to multipath reflection.

[0045] The circuit can be configured to obtain model-based information through a ray tracing process. For example, in 3D reconstruction, the normal vector field of the vehicle surface can be calculated. From the normal vector field, the orientation of the surface can be known, which can be used as a basis for determining light reflection.

[0046] The model-based information can include predicted photon arrival times.

[0047] The predicted photon arrival times can be related to at least one of second-order light reflection, third-order light reflection, and higher-order light reflection.

[0048] The model-based information can include predictions of multipath light reflection.

[0049] The circuit can be configured to determine whether the ToF information obtained from the reflected light deviates from the model-based information. For example, the model-based predicted positions of the peaks of primary reflection, secondary reflection, and subsequent reflections in the photon histogram can be compared with the corresponding positions of the measured peaks in the ToF histogram. Here, the positions in the photon histogram can correspond to the arrival times of the photons.

[0050] For example, if the comparison result is within a predefined threshold, it can be determined that the parking space is empty. However, if it is determined that the comparison result is outside the threshold, it can be determined that the parking space is not empty, i.e., occupied or blocked.

[0051] The circuit can be configured to determine whether the position of the reflection indicated by the ToF information deviates from the model-based prediction.

[0052] For the case of using a dToF camera, based on the predicted arrival time of the reflected light, the peak in the histogram measured by the dToF camera is determined and compared with the model prediction.

[0053] For another case of using iToF, the circuit can be configured to determine whether the measured iToF phasor is affected by scattering due to a long reflection (e.g., an empty parking space) and thus deviates from the model-based phasor prediction, which is based on the predicted arrival time of the reflected light.

[0054] The circuit can be configured to perform model-based light reflection prediction based on a reconstructed 3D model of the captured scene.

[0055] Associating depth information obtained from ToF measurements with a reconstruction model of a scene (i.e., a running 3D reconstruction) may include any processing performed on the raw ToF measurements, such as processing the raw measurements obtained from sensors in the ToF data path. Associating depth information obtained from ToF measurements with the reconstruction model may also include: transforming the ToF measurements into a point cloud, registering the point cloud to the reconstruction model, etc. The circuit may be configured to reconstruct and / or update a model of the scene based on depth information obtained from ToF measurements. The model of the scene may be updated, for example, based on point cloud information and / or registered point cloud information.

[0056] The circuit may be configured to perform model-based light reflection prediction based on one or more vehicle models.

[0057] For example, vehicles within the field of view of a camera may be identified. This identification of a vehicle or other object may use a point cloud or an RGB or grayscale image as input, and 3D reconstruction techniques and machine learning algorithms classify regions of the image, portions of the point cloud, or portions of the reconstructed 3D model as a vehicle.

[0058] The modeled surface (and its orientation) of the vehicle included in the vehicle model may be used in a ray tracing-type process to calculate the intersection points of light beams, thereby determining the locations where the light beams are partially reflected and partially scattered. Based on the generated light beams, the distances traveled by the light of secondary reflections and subsequent reflections to reach those reflection and scattering points and return to the ToF camera system may be calculated.

[0059] The vehicle model may model components of the vehicle that are not visible to the ToF imaging sensor in the imaging information obtained from the primary reflection.

[0060] For example, when capturing a point cloud from the position of the ToF camera system, there are shadow regions where an object hides other objects. Additionally, the opposite side or the back side of an object is typically not captured. According to an embodiment, the vehicle surfaces not included in the point cloud captured by the sensor are modeled.

[0061] The circuit may be configured to determine a vehicle model based on a 3D model of the scene.

[0062] The vehicle model may model components of the vehicle that do not exist in the 3D model of the scene.

[0063] The circuit may be configured to emit light and obtain ToF information from at least one of second-order reflections, third-order reflections, and higher-order reflections of the emitted light.

[0064] The circuit may be configured to emit light, and wherein the circuit is configured to model the optical path of the emitted light to obtain model-based light reflection prediction.

[0065] The embodiment also discloses a method, including: determining the state of a parking space based on a comparison between ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection.

[0066] The embodiment also discloses a computer-implemented method and / or a computer program, including instructions that are configured to, when executed on a processor, perform the above method and process.

[0067] The embodiment also discloses a computer program product, including instructions that are configured to, when executed on a processor, perform the above method and process.

[0068] Figure 1 A vehicle having a driver assistance system for parking space detection and a ToF camera is schematically shown.

[0069] Vehicle 100 including a driver assistance system for parking space detection approaches three parked vehicles 101, 102, and 103. An empty parking space is located between vehicles 102 and 103.

[0070] The ToF camera 110 of the driver assistance system has a field of view 105, and vehicles 101, 102, and 103 are within the field of view 105. A line of sight 111 tangent to vehicle 102 delimits an invisible area 120 of the parking space. Due to vehicle 102, the driver of the ToF camera system 110 and vehicle 100 cannot see a portion 120 of the parking space located above the line of sight 111. The ToF camera 110 can be, for example, an iToF (indirect time-of-flight) camera or a dToF (direct time-of-flight) camera.

[0071] Due to the invisible area 120, neither the ToF camera system 110 nor the driver of vehicle 100 can determine via a direct line of sight whether the parking space between vehicles 102 and 103 is empty or whether a vehicle or any other object is blocking the parking space in the invisible area 120. Therefore, from the perspective of the ToF camera system 110 and the driver of vehicle 100, the parking space may be empty and will hereinafter be referred to as a potentially empty parking space.

[0072] Driver assistance system for detecting non-line-of-sight objects

[0073] According to the embodiment described below, a driver assistance system is provided that detects non-line-of-sight (NLOS) objects based on a comparison between ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection. The embodiment described below, for example, relates to detecting non-line-of-sight objects using histogram analysis in an automotive environment.

[0074] For example, a driver assistance system can be configured to detect at least one of the presence, location, and speed of NLOS objects. By detecting NLOS objects, the driver assistance system can, for example, determine the status of a point (e.g., a parking space). For example, the status of the point determined by the driver assistance system can include information about whether the point is empty.

[0075] Figure 2 An example of the configuration of a driver assistance system for parking space detection is schematically shown.

[0076] Scene 201 (e.g., the scene of vehicles 101, 102, 103 described above Figure 1 is illuminated by a ToF camera 202 (see also Figure 2 ), and the reflected light from scene 201 is captured by the ToF camera 202. The ToF camera 202 can be, for example, an iToF (indirect time-of-flight) camera or a dToF (direct time-of-flight) camera.

[0077] The ToF camera 202 includes a ToF camera controller 202-1 that controls the operation of the camera's illuminator and sensor according to a configuration mode that defines configuration settings (such as exposure time, etc.) related to the operation of the imaging sensor and the lighting sensor. The controller 202-1 provides ToF measurements (e.g., SPAD histograms, iToF phase diagrams) to a ToF data path 202-2 that processes the ToF measurements into a ToF point cloud (e.g., defined in a 3D camera coordinate system). The ToF point cloud is a point representation of the ToF measurements that describes the current scene as seen by the ToF camera. The ToF point cloud can be represented, for example, in the Cartesian coordinate system of the ToF camera. This ToF point cloud obtained from the ToF data path is forwarded to 3D reconstruction 204.

[0078] The 3D reconstruction 204 is based on techniques known to those skilled in the art (e.g., based on the following regarding Figure 16A more detailed description of the technology) is used to create and maintain a three-dimensional (3D) model of scene 201. Specifically, 3D reconstruction 204 includes receiving pose estimation 204-1 of the ToF point cloud. Pose estimation 204-1 also receives auxiliary input from auxiliary sensor 203 and the current 3D model from 3D model reconstruction 204-2. Based on the ToF point cloud, auxiliary input, and the current 3D model, pose estimation 204-1 applies an algorithm to the measurements to determine the pose (defined by, for example, position and orientation) of the ToF camera in the global scene ("world"). Such algorithms can include, for example, the iterative closest point (ICP) method between the point cloud information and the current 3D model, or, for example, a SLAM (simultaneous localization and mapping) pipeline. After knowing the camera pose, pose estimation 204-1 "registers" the ToF point cloud obtained from data path 202-2 into the global scene, thereby generating a registered point cloud, which represents the transformation of the point cloud in the camera coordinate system to the global coordinate system (e.g., the "world" coordinate system) in which the model of the scene is defined.

[0079] The registered point cloud obtained through pose estimation 204-1 is forwarded to 3D model reconstruction 204-2. 3D model reconstruction 204-2 updates the 3D model of the scene based on the registered point cloud obtained from pose estimation 204-1 and the auxiliary input obtained from auxiliary sensor 203. An exemplary process for updating the 3D model is described in more detail below with respect to Figure 16 An exemplary process for updating the 3D model is described in more detail below with respect to

[0080] The updated 3D model of scene 201 can be stored in a 3D model memory ( Figure 2 not shown) and provided to NLOS object detection 205. NLOS object detection 205 can determine the status of a parking space, for example, based on the presence of NLOS objects within the parking space.

[0081] As described above, pose estimation 204-1 and 3D model reconstruction 204-2 obtain auxiliary input from auxiliary sensor 203. Auxiliary sensor 203 includes a color camera 203-1 that provides, for example, RGB / LAB / YUV images of scene 201 from which sparse or dense visual features can be extracted to perform traditional visual odometry, i.e., to determine the position and orientation of the current camera pose. Auxiliary sensor 203 can also include an event-based camera 203-2 that provides, for example, high-frame-rate cues for visual odometry based on events. Auxiliary sensor 203 can also include an inertial measurement unit (IMU) 203-3 that provides, for example, acceleration and orientation information that can be appropriately integrated to provide pose estimation. These auxiliary sensors 203 collect information about scene 201 to assist 3D reconstruction 204 in generating and updating the 3D model of the scene (see Figure 3 and the corresponding description).

[0082] The above-mentioned Figure 2 The ToF camera 202 and the auxiliary sensor 203 described in, for example, may be part of an out-of-vehicle information detection unit including an imaging unit ( Figure 17 7410 in) and a vehicle exterior information detection unit ( Figure 17 7400 in). 3D reconstruction 204 and parking space detection 205 may be implemented in one or more processors (e.g., a processor of a vehicle control system such as Figure 17 the integrated control circuit 7600 in).

[0083] It should be noted that in the Figure 2 embodiment, the parking space detection is only used as an example of detecting non-line-of-sight objects by comparing the ToF information obtained from the reflected light received by the ToF imaging sensor with the model-based information obtained from the model-based light reflection prediction. In other embodiments, the detection of non-line-of-sight objects is used to observe, for example, oncoming vehicles or objects on a perpendicular street that cannot be directly seen by the vehicle. In addition, a secondary histogram peak such as a Kalman filter can be used to track the absolute speed of an object to, for example, determine whether the vehicle collides with an object in NLOS.

[0084] Other possible automotive applications for detecting non-line-of-sight objects are detecting lane change space.

[0085] The technology disclosed in the embodiments can also be applied to non-automotive environments. For example, the technology can be used to identify the inventory of spaces such as warehouses (using robots, drones), or for transportation (drones need to find dedicated empty spots).

[0086] ToF data path

[0087] The ToF data path ( Figure 2 202-2 in) is configured to receive camera raw data (ToF measurements) and further process the raw data, for example, to process it into a ToF point cloud (e.g., defined in a 3D camera coordinate system). The ToF data path may, for example, include iToF-related processing that generates a depth frame for 3D reconstruction from iToF sensor data. Further, the ToF data path may include processing related to generating a histogram for NLOS detection from dToF sensor data.

[0088] The ToF data path may also perform processing such as transforming the depth frame into a vertex map and a normal vector.

[0089] The ToF data path may also include a sensor calibration block that eliminates phase, systematic error sources (such as temperature drift), cyclic errors due to spectral aliasing on the returned signal, and any errors due to electrical non-uniformity of the pixel array through calibration. According to Equation 1, from the measurements q0, …, q at the pixel N the obtained phase value φ, the corresponding depth value D of the pixel is determined as follows:

[0090]

[0091] where f mod is the modulation frequency of the transmitted signal, and c is the speed of light.

[0092] For each frame k, based on the depth measurement D of each pixel k , the three-dimensional coordinates in the camera coordinate system are determined, thereby obtaining the ToF point cloud of the current frame k. In addition, the ToF data path 202-2 may include filters that improve signal quality and reduce errors on the point cloud, such as ToF data denoising, removing pixels that are incompatible with the viewing point (e.g., “flying” pixels between the foreground and background), and removing multipath effects such as scene, lens, or sensor scattering.

[0093] 3D Reconstruction

[0094] Figure 1 The 3D reconstruction 204 of receives the ToF point cloud and generates a 3D model of the scene 201 while tracking the motion of the ToF camera (i.e., the current pose of the ToF camera). Those skilled in the art also refer to this problem as “simultaneous localization and mapping”. There are several methods to solve this problem, such as SLAM based on the extended Kalman filter, parallel tracking and mapping, etc. For example, an overview of different SLAM methods is given in the paper “Past, Present, and Future of Simultaneous Localization and Mapping: Towards the Robust-Perception Age” by C. Cadena et al., in the IEEE Transactions on Robotics, Vol. 32, No. 6, pp. 1309-1332, 2016.

[0095] Auxiliary sensor data (e.g., from Figure 2 the auxiliary sensor 203) may optionally be used at several stages to improve 3D model reconstruction. The main use may be to provide additional data streams that can be used to improve or optimize the quality of pose estimation by fusing different cues and complementary features in the sensor data( Figure 1in 204-1). For example, extracting sparse features from RGB frames can be used to perform visual odometry by finding feature correspondences in consecutive frames. Thus, sensor data can be jointly used to estimate a single pose in pose estimation (e.g., ICP method or SLAM pipeline). The auxiliary sensor unit and the ToF system can operate in a sensor fusion camera suite for specific target use cases.

[0096] The 3D model reconstruction 204-2 provides the updated 3D model to the parking space detection 205 of the driver assistance system.

[0097] Figure 3 Schematically shows an example of a point cloud generated by the ToF data path of a driver assistance system for parking space detection in a parking lot.

[0098] The point cloud 301 depicts a parking lot with parking spaces and pillars on the left and right sides, as well as a central lane 305. The point cloud can be obtained, for example, through the ToF data path of the driver assistance system for parking space detection as described above Figure 2 in (see Figure 2 202-2 in).

[0099] Depicts two vehicles 302 and 304 and a pillar 303 between the two vehicles 302 and 304. The distance between the two vehicles 302 and 304 is large enough for a potential empty parking space. However, both the vehicle 304 and the pillar 303 block the line of sight to the potential empty parking space. There is no (direct) line of sight to determine whether the parking space is really empty.

[0100] Based on Figure 3 this point cloud shown in, the 3D model reconstruction ( Figure 2 204-2 in) can determine the 3D model of the scene (here a parking lot) and provide this 3D model to the parking space detection ( Figure 2 205 in).

[0101] 3D model

[0102] Figure 4 Shows an example of a 3D model of the scene details generated by 3D reconstruction such as described above Figure 2 in.

[0103] The 3D model is implemented as a triangle mesh 401. The triangle mesh can be a local or global three-dimensional triangle mesh. In alternative embodiments, the 3D model can also be described by a local or global voxel representation of the point cloud (uniform or octree); a local or global occupancy grid; a mathematical description of the scene based on planes, statistical distributions (e.g., Gaussian mixture model), or similar properties extracted from the measured point cloud.

[0104] In another embodiment, the model can be characterized as a mathematical object that satisfies one or more of the following aspects: it can be projected onto any arbitrary view, it can query the nearest neighbor (the closest model point) relative to any input 3D point, it calculates the distance relative to any 3D point cloud, it estimates the normal, and / or it can be resampled at arbitrary 3D coordinates.

[0105] The model can be implemented, for example, as a triangular mesh (e.g., a local or global three-dimensional triangular mesh), a local or global voxel representation of a point cloud (uniform or octree), a local or global occupancy grid, a mathematical description of a scene based on planes, statistical distributions (e.g., Gaussian mixture models), or similar properties extracted from measured point clouds. The model is typically built up step by step by fusing measurements from available data sources, such as, including but not limited to, depth information, color information, inertial measurement unit information, event-based camera information.

[0106] Based on the surface elements (e.g., the elements of the mesh) defined by the 3D model, the normal vector of the surface element can be calculated, and this normal vector can be used in a ray tracing process (see Figure 11 ) etc. for parking space detection in the embodiments described in more detail below.

[0107] Classification techniques are known to those skilled in the art and are used to construct the detected scene into different objects, such as, for example, cars, street markings, walls, etc. These classification techniques can be, for example, pattern matching and may be based on manual feature extraction, such as histograms of oriented gradients. Additional techniques can use convolutional neural networks, general deep learning, or "you only look once" classifiers.

[0108] Multipath interference (MPI)

[0109] It is well known that the light reflected from a scene includes so-called multipath interference (MPI). Multipath interference (MPI) is caused by multiple paths from the illumination source to the same pixel. MPI is typically one of the most important error sources in ToF depth measurements. However, as explained with reference to the embodiments described in more detail below, the information from multipath interference can be used in a beneficial way to enhance the evaluation of the detected scene. Specifically, the embodiments described below evaluate the information obtained from multipath interference caused by the reflection of light on the surface of an object.

[0110] Parking space detection

[0111] Based on the 3D model provided by the 3D model reconstruction (e.g., Figure 2 in 204 - 2), parking space detection ( Figure 2 in 205) can determine the relevant surface, parking orientation, and the distance between vehicles (e.g., Figure 1 102, 103 in

[0112] Figure 5 Schematically shows the line of sight of a driver assistance system for parking space detection. Depicted is the same parking space detection situation as described in Figure 1 the same.

[0113] The ToF camera 110 of the driver assistance system has a field of view 105, and the vehicles 101, 102, and 103 are within the field of view 105. The line of sight 111a intersects the vehicle 101 within the field of view 105 of the ToF camera 110 and helps to determine the surface model of the vehicle 101. In addition, the line of sight 111b intersects the vehicle 102 and helps to determine the surface model of the vehicle 102. Since the line of sight 111b intersects the vehicle 102 and is blocked by the vehicle 102, it does not reach the invisible area 120 in the empty parking space between the vehicles 102 and 103. In addition, the lines of sight 111c and 111d intersect the vehicle 103 and help to determine the surface model of the vehicle 103. The line of sight 111c hits the vehicle 103 in the region of interest 115 (ROI) on the vehicle 103.

[0114] Reference Figure 6 explains further the path of such secondary reflections and subsequent reflections.

[0115] Figure 6 Schematically shows a vehicle with a driver assistance system for parking space detection based on measurements of multipath reflections when the parking space is empty.

[0116] Figure 5 Depicts a parking space detection situation, where the illumination and back-reflection, back-scattering beam paths are indicated by the arrows 106, 107, and 108.

[0117] Within the ROI 115, the illumination beam 106 hits the surface of the vehicle 103, and a part of the illumination beam 106 is reflected according to the law of reflection, where the incident beam 106 and the reflected beam 107 have the same angle with the surface normal 112 of the vehicle surface in the ROI 115. A part of the illumination beam 106 is scattered at the surface of the vehicle 103 in the ROI 115, returns to the ToF camera 110 (along the path of the illumination beam 106), is captured by the sensor of the ToF camera 110, and produces a peak at the position corresponding to the travel distance of the light in the ToF histogram (see Figure 9a 911 in

[0118] The reflected light beam 107 hits the surface of vehicle 102, and a part of the reflected light beam 107 is reflected according to the law of reflection to generate a reflected light beam 108. A part of the illumination light beam 107 scatters at the surface of vehicle 102, returns to the ToF camera 110 (along the paths of the reflected light beam 107 and the illumination light beam 106), is captured by the sensor of the ToF camera 110, and generates another peak at a position corresponding to the travel distance of the light in the ToF histogram (see Figure 9a 912 in

[0119] The reflected light beam 108 hits the surface of vehicle 103, and a part of the reflected light beam 108 scatters at the surface of vehicle 102, returns to the ToF camera 110 (along the paths of the reflected light beam 108, the reflected light beam 107 and the illumination light beam 106), is captured by the sensor of the ToF camera 110, and generates another peak at a position corresponding to the travel distance of the light in the ToF histogram (see Figure 9a 913 in

[0120] As in Figure 6 the invisible region 120 corresponds to an empty parking space, and the light beams 106, 107 and 108 can traverse the invisible region 120 unobstructed.

[0121] Otherwise, if the space between vehicles 102 and 103 is not an empty parking space but is occupied by another vehicle, the light beams 106, 107 and 108 cannot traverse the invisible region 120 unobstructedly and will be blocked by the other vehicle.

[0122] The paths of the light beams along arrows 106, 107 and 108 are modeled and measured. When modeling the light beams along arrows 106, 107 and 108, the shape parts of vehicles 102 and 103 can be assumed by identifying, for example, the make and model and filling in the blanks of a predefined model with the same orientation as the measured vehicle.

[0123] Figure 7a A first example of a vehicle model of a parked vehicle derived from a reconstructed 3D model by a driver assistance system for parking space detection is schematically shown.

[0124] In Figure 7a a surface model M1 of the first vehicle ( Figure 2 102 in Figure 6 ) determined by reconstructing from the 3D model ( Figure 6 204-2 in

[0125] Based on the models M1 and M2 of the visible surfaces of the vehicle, corresponding vehicle models VM1 and VM2 are created. The vehicle models VM1 and VM2 are models of the complete vehicle shape including the invisible surfaces of the vehicle. That is, for the vehicle models VM1 and VM2, the surfaces F1 and F2 facing each other between the two vehicle models VM1 and VM2 are known. The surface F1 can correspond, for example, to the surface of the first vehicle ( Figure 6 in 102) facing the empty parking space (see Figure 6 ), and the surface F2 can correspond, for example, to the surface of the second vehicle ( Figure 6 in 103) facing the empty parking space.

[0126] In Figure 7a 's implementation, the vehicle models VM1 and VM2 are schematic box-shaped models providing a simplified geometry of the vehicle. The dimensions of the corresponding sides of the box correspond to the dimensions of the vehicle as determined from the surface models M1 and M2 of the vehicle. The vehicle models can also include information about the position and orientation of the vehicle, for example, in a global coordinate system.

[0127] Figure 7b Schematically shows a second example of a vehicle model of a parked vehicle derived from a reconstructed 3D model by a driver assistance system for parking space detection.

[0128] Depicts the positions of the wheels W1 and W2 of the first vehicle ( Figure 2 in 102) as determined by the 3D model reconstruction ( Figure 6 in 204 - 2). The axis A1 from the center of the wheel W1 to the center of the wheel W2 is determined. The surface F1 of the vehicle model VM1 is derived from the axis A1. The surface F1 is determined as the surface perpendicular to the axis A1 and intersecting the outer surface of the wheel W2. Further depicts the positions of the wheels W3 and W4 of the second vehicle ( Figure 2 in 103) as determined by the 3D model reconstruction ( Figure 6 in 204 - 2). The axis A2 from the center of the wheel W3 to the center of the wheel W4 is determined. The surface F2 of the vehicle model VM2 is derived from the axis A2. The surface F2 is determined as the surface perpendicular to the axis A2 and intersecting the outer surface of the wheel W3.

[0129] In this way, the surface F1 intersects the wheel W2 on the side facing away from the wheel W1 and towards the vehicle model VM2, and the surface F2 intersects the wheel W3 on the side facing away from the wheel W4 and towards the vehicle model VM1. The surface F1 can correspond, for example, to the surface of the first vehicle ( Figure 6 in 102) facing the empty parking space (see Figure 6 ), and the surface F2 can correspond, for example, to the surface of the second vehicle ( Figure 6 in 103) facing the empty parking space.

[0130] Figure 8 Schematically shows a process of determining the travel time of an illumination beam reflected between two vehicles by ray tracing based on a vehicle model that is an object in a scene.

[0131] Figure 8 The process described in is a schematic description of ray tracing techniques well known to those skilled in the art and is often used in 3D computer graphics. In this example, ray tracing is used as a technique for modeling light transport and in a rendering algorithm that implements a driver assistance system for parking space detection as described above.

[0132] In Figure 8 depicts vehicle models VM1 and VM2 derived from the process with reference to Figure 7a or Figure 7b . These vehicle models VM1 and VM2 are models of the complete vehicle shape including the invisible surfaces of the vehicle. That is, for vehicle models VM1 and VM2, the surfaces F1 and F2 facing each other between the two vehicle models VM1 and VM2 are known. Surface F1 corresponds to the surface of the first vehicle ( Figure 6 102 in Figure 6 ) facing the empty parking space (see Figure 6 ), and surface F2 corresponds to the surface of the second vehicle (

[0133] point P ToF is the position of the ToF camera (see Figure 6 110 in ROI is the position of the ROI (see Figure 6 115 in

[0134] Starting from the position P ToF of the illuminator in the ToF camera, the illumination light emitted by the ToF camera travels along the optical path l1 ("main ray") to the position P ROI where the illumination beam hits surface F2. The direction of the optical path l1 can be defined, for example, by the corresponding pixel of the ToF sensor that corresponds to the region of interest ROI.

[0135] At P ROI the illumination light hitting surface F2 (by diffuse and specular reflection) generates reflected light. A part of this reflected light travels back along the optical path l1 to the ToF camera and generates an event in the pixel of the image sensor at an arrival time depending on the travel time (determined by the travel distance along the optical path l1 and the speed of light). Another part of the illumination light hitting surface F2 at point P ROI is at point P ROIis specularly reflected and continues along the optical path l2 (“secondary light beam”). According to well-known optical principles, the direction of the optical path l2 depends on the surface normal N1 of the surface F2 at the point P ROI and the angle of incidence α of the optical path l1 on the surface F2. The optical path l2 is traced back to the position P1 where it impinges on the surface F1.

[0136] The illumination light that impinges on the surface F1 at P1 generates reflected light (through diffuse and specular reflection). A part of this reflected light returns to the ToF camera along the optical paths l2 and l1 and generates an event in the pixels of the image sensor at an arrival time that depends on the travel time (as determined by the travel distances along the optical paths l1 and l2 and the speed of light). Another part of the illumination light that impinges on the surface F1 at the point P1 is specularly reflected at the point P1 and continues along the optical path l3 (“tertiary light beam”). The direction of the optical path l3 depends on the surface normal N2 of the surface F1 at the point P1 and the angle of incidence α of the optical path l2 on the surface F1. The optical path l3 is traced back to the position P2 where it impinges on the surface F2.

[0137] The illumination light that impinges on the surface F2 at P2 generates reflected light (through diffuse and specular reflection). A part of this reflected light returns to the ToF camera along the optical paths l3, l2, and l1 and generates an event in the pixels of the image sensor at an arrival time that depends on the travel time (as determined by the travel distances along the optical paths l1, l2, and l3 and the speed of light).

[0138] At this stage, when a predefined maximum number of reflections is reached, the recursive ray tracing process ends.

[0139] With this ray tracing technique, in the case of a dToF camera, multiple arrival times of light are predicted, which appear as peaks in the dToF photon histogram obtained from the pixels of the dToF camera.

[0140] Since in the further processing of the histogram (see Figure 9a and the corresponding description), only the arrival times of the photons are evaluated, the amount of light collected by the pixel is not necessary, such that the ray tracing algorithm does not necessarily need to consider details such as determining the intensity of light according to the principles of specular reflection (e.g., Phong shading) and diffuse reflection (e.g., according to Lambertian shading).

[0141] Generally speaking, in ray tracing, each pixel of the sensor can define a corresponding primary ray, and each primary ray of the image can be traced and used to detect non-line-of-sight objects. To reduce the computational workload, a region of interest (ROI) can be selected (see also above Figure 5 and Figure 6), and performing the ray tracing process and photon histogram analysis may be limited to the primary rays related to the ROI (e.g., hitting the ROI). For example, the region of interest ROI can be selected based on the reflectivity and normal vectors of the pixels of the car 103 in the FOV. In other embodiments, each pixel can be marked using the above classifier to identify the region defining the vehicle body (i.e., potential ROI candidate pixels). For example, those pixels classified as tires can be ignored, but pixels related to the car chassis can be included in the ROI. The selection of the ROI has the constraint that non-specular reflection points must be selected for iToF and dToF measurements.

[0142] For example, in Figure 5 the parking space detection shown, the reflectivity in the ROI should preferably be such that it is maximum for the car (which means that the light will bounce back with a very high intensity on highly specular materials rather than on the tires), and the normal vectors of the selected pixels should cause the secondary reflection to intersect with the car 102 or an object in the field of view (such as Figure 3 the pillar in Figure 10 or other obstacles such as the car 104 in Figure 5 ). In this way, when determining the ROI, those rays that are more likely to contain strong secondary or subsequent peaks are selected, thus producing a good signal-to-noise ratio, for example, good histogram features in dToF or high-scattering phasors in iToF. An example of such a region can be the 10×10 pixel region on the rear door of the car 103 in Figure 5 . Conversely, reflections that do not intersect with the car 102 (e.g., the primary ray reflected back from the rear of the car 103 to the street) or reflections that are otherwise unavailable (e.g., the primary ray hitting the black tire) may be discarded.

[0143] In the case of using an iToF camera as the ToF camera, multiple predicted arrival times of light can be used to calculate and thus predict multiple phasors of the iToF measurement. The phase between the modulated light signal emitted by the iToF camera for illuminating the vehicle ( Figure 5 102 and 103 in

[0144] along the optical path ( Figure 8 l1 in Figure 9b 921 in

[0145] In addition, the modulated reflected light returning to the iToF camera along the optical paths l1 and l2 has a phase difference with the emitted light, which depends on the travel time (such as determined by the travel distance along the optical paths l1 and l2 and the speed of light). Based on this phase difference, the phasor in the phasor diagram can be calculated ( Figure 9b 922 in the above).

[0146] Therefore, if the point (parking space) is empty, the above prediction of the arrival time of the reflected light can be used to predict the measurement results of the iToF camera and the dToF camera.

[0147] Figure 9a An example of predicted photon arrival times and a measured dToF histogram for the case where a parking space is empty is schematically shown.

[0148] The dToF histogram 900 depicts predicted arrival times (dashed line) and measured ToF signals (solid line) over time. The dToF histogram 900 is measured if the ToF camera is a direct ToF camera.

[0149] ToF histogram 900 depicts the Figure 8 The ray tracing process predicts the primary reflection ( Figure 8 The ToF histogram 900 further depicts the prediction of the arrival time of the photon corresponding to the light path l1) traveling twice in the light path l1. Figure 8 The ray tracing process predicts secondary reflections ( Figure 8 The ToF histogram 900 further depicts the prediction of the arrival time of the photon corresponding to the light path l1 and l2 in the image. Figure 8 The ray tracing process predicts the three-level reflections ( Figure 8 903 prediction of the arrival time of the photon corresponding to the light path l1, l2 and l3 traveled twice.

[0150] Figure 9a It also depicts Figure 6 As shown in the case of the ToF camera ( Figure 6 The captured peak values of the primary reflection 911, secondary reflection 912 and tertiary reflection 913 captured by 110).

[0151] exist Figure 9a In the example of , the measured peaks 911, 912 and 913 are located at the corresponding predicted arrival times 901, 902 and 903. Figure 9a In the example of FIG. 1 , the fact that the measured peaks 911 , 912 , and 913 are located at the corresponding predicted arrival times 901 , 902 , and 903 is based on the following aspects: Figure 9a The histogram refers to Figure 6 In the case that the parking space is empty, the light beam can be seen in the vehicle (seeFigure 6 travel freely between the surfaces of 102 and 103 in, which is also the Figure 8 assumption made in the ray tracing algorithm described in.

[0152] Figure 9b Schematically shows an example of the predicted phasor and the measured iToF phasor diagram when the parking space is empty.

[0153] Depicts the measured Figure 6 and Figure 8 phasor diagram of the situation in. The iToF phasor diagram 920 depicts the predicted phase (dashed line) derived from the predicted time of arrival ( Figure 8 ) and the measured phasor (solid arrow) derived from the phase of the iToF signal as measured in Figure 6 .

[0154] Since the iToF camera can only measure one phase, the measured phase Φ is the phase of the phasor 914 derived by vectorially adding the phasors 911 and 912 of the reflected signals of the primary reflection and the secondary reflection.

[0155] The phase 921 shown by the dashed line can be predicted based on the time delay between the received modulated photon signal and the modulated photon signal transmitted by the iToF camera. The phase 921 corresponds to the primary reflection predicted by the ray tracing process of Figure 8 ( Figure 8 traveling twice the optical path l1). The time delay between the transmitted photon and the photon arrival time and the frequency of the modulated photon signal emitted by the iToF camera can be converted into the phase 921 shown by the dashed line. If the primary reflection ( Figure 8 traveling twice the optical path l1) can be measured independently of the secondary reflection ( Figure 8 traveling twice the optical path l1 and l2), then the dashed line of the phase 921 represents the orientation of the phasor.

[0156] In addition, the phase 922 shown by the dashed line can be predicted based on the time delay between the received modulated photon signal and the modulated photon signal transmitted by the iToF camera. The phase 922 corresponds to the secondary reflection predicted by the ray tracing process of Figure 8 ( Figure 8 traveling twice the optical path l1 and l2). The phase 922 can be determined in the same way as the phase 921. If the secondary reflection ( Figure 8 traveling twice the optical path l1) can be measured independently of the primary reflection ( Figure 8 traveling twice the optical path l1 and l2), then the dashed line of the phase 922 represents the orientation of the phasor.

[0157] Figure 9bAlso shown is the measured phasor 914, which, if measurable separately, could be derived by vector addition of the phasors 911 and 912 of the primary reflection and the secondary reflection in the case shown in Figure 6 . Since the phasors 911 and 912 cannot be measured separately, they are depicted as dashed arrows and are only used to explain the derivation of the measured phasor 914. Additionally, the phases of 911 and 914 can be inferred (predicted) from the surfaces F1 and F2 in Figure 7a . To calculate the positions of F1 and F2, components least affected by multipath (e.g., wheels) can be detected, their positions in space calculated, and the positions of the surfaces inferred based on, for example, a vehicle model. Figure 9b

[0158] The phasor 915 corresponds to the phasor 912, but its starting point is located at the end point of the phasor 911. The starting point of the phasor 911 and the end point of the phasor 915 correspond to the starting point and the end point of the measured phasor 914, representing the vector addition of the phasors 911 and 912.

[0159] When the parking space is empty, the phase Φ of the measured phasor 914 must be assigned to the measurement signals of the primary reflection and the secondary reflection as shown in Figure 6 and Figure 8 .

[0160] The phasors 911 and 912 are located at the corresponding predicted phasors 921 and 922. In the example of Figure 9b , the fact that the phasors 911 and 912 are located at the corresponding predicted phasors 921 and 922 is based on the following: in the case of the phasor diagram of Figure 9b referring to Figure 6 , the parking space is empty, such that the light beam can travel freely between the surfaces of the vehicle (see 102 and 103 in Figure 6 ), which is also the assumption made in the ray tracing algorithm described in Figure 8 . Here, being located at the corresponding predicted phasors 921 and 922 means having the same phase as the corresponding predicted phasors 921 and 922.

[0161] In the example of Figure 9b , the measured phasor 914 deviates from the predicted phase 921 of the primary reflection, so it can be assumed that the light beam can travel freely through the optical path (107 in Figure 6 and l2 in Figure 8 ), and the parking space is empty. If the light beam cannot travel freely through the optical path (107 in Figure 6 and l2 in Figure 8 ) and the parking space is not empty, then the deviation of the measured phasor 914 from the predicted phase 921 is smaller, as will be referenced in Figure 12b ​as discussed. This deviation can be classified using a threshold or a hypothesis test, for example, as described in reference Figure 19 as described.

[0162] For the measurement and evaluation method of such a light beam, when the parking space is empty, the scattering caused by multipath is more important. In the case where the parking space is occupied, a higher intensity corresponding to the norm of phasor 912 is expected (inverse square law of the traversal distance of the light beam), but the phase difference between phasor 911 and phasor 912 is minimal because Figure 11 the distance 107 in Figure 11 is lower than the distance 107 in Figure 9a ). The selection of the ROI should be the same as in the case of using a dToF camera ( Figure 9a ), but the limitation is that a non-specular reflection point must be selected. The selection of the ROI is very important to detect the features of hidden objects inside the iToF phasor map of the pixels most affected by multipath. In fact, 912 is generated not only by l1, but also by l2 and l3 and all subsequent reflections. However, due to the inverse square law of the traversal distance of the light beam, the contribution of subsequent reflections becomes smaller and more negligible.

[0163] In contrast, reference Figure 10 explains the case of detecting a parking space where a potentially empty point is actually not empty.

[0164] Figure 10 Schematically shows a vehicle with a driver assistance system and a ToF camera for parking space detection when the parking space is not empty.

[0165] Depicts Figure 1 the parking space detection situation. However, the potentially empty parking space between vehicles 102 and 103 is blocked by the vehicle 104 parked further forward. Neither the driver of vehicle 100 nor the ToF camera 110 has a direct line of sight to vehicle 104. This is indicated by the fact that the line of sight 111 tangent to vehicle 102 does not intersect vehicle 104. Thus, vehicle 102 blocks the line of sight of each object in the invisible area 120 above the line of sight 111.

[0166] This may cause the driver to think that the parking space may be empty, while in fact it is occupied by another parked car. Therefore, the driver assistance system for parking space detection needs to determine that the parking space is not empty.

[0167] Figure 11 The measurement situation of whether the parking space is empty is depicted in

[0168] Figure 11 Schematically shows a vehicle with a driver assistance system for parking space detection based on measurements of multipath reflections when the parking space is not empty.

[0169] Figure 11 The scenario corresponding to Figure 6 the scenario. However, except that Figure 6 the parking space between vehicles 102 and 103 in Figure 6 is empty, vehicle 104 is parked in the parking space between vehicles 102 and 103. The illumination beam 106 (primary beam) is reflected at the surface of vehicle 103 in ROI 115 and generates a reflected beam 107 (secondary beam). The beam 107 hits vehicle 104 (instead of

[0170] vehicle 102 in Figure 8 .

[0171] Figure 12a Schematically shows an example of the predicted photon arrival time and the measured dToF histogram in the case where the parking space is not empty. This dToF histogram 1200 is measured when the ToF camera is a direct ToF camera.

[0172] Figure 12a The prediction of the arrival time in Figure 8 is based on the assumption that the parking space is empty and the light beam ( Figure 9a the optical paths l2 and l3 in

[0173] ) can freely traverse the empty parking space. Therefore, the predicted peak is the same as Figure 8 in Figure 8 . Figure 8 So, the ToF histogram 1200 depicts the prediction 901 of the photon arrival time corresponding to the primary reflection ( Figure 8 traveling twice the optical path l1 as predicted by the ray tracing process of Figure 8 . The ToF histogram 900 further depicts the prediction 902 of the photon arrival time corresponding to the secondary reflection ( Figure 8 traveling twice the optical paths l1 and l2 as predicted by the ray tracing process of

[0174] Figure 12a . The ToF histogram 900 further depicts the prediction 903 of the photon arrival time corresponding to the tertiary reflection ( Figure 11 traveling twice the optical paths l1, l2 and l3 as predicted by the ray tracing process of Figure 11 ).

[0175] The peak of the primary reflection 911 matches the prediction 901 of the photon arrival time. However, the captured peak of the secondary reflection 912 is measured at an earlier time than the prediction 902 of the photon arrival time.

[0176] This is because the distance traveled by the diffusely reflected light beam 107 at the vehicle 104 is shorter than the predicted optical path (see l1 and l2 in Figure 8 . Therefore, it is determined that the parking space is not empty. This can also be determined by the peak of the captured tertiary reflection (not shown) to predict the photon arrival time 903 corresponding to the tertiary reflection, or all other reflections can be determined if these reflections are predicted and measurable.

[0177] In addition, to determine whether the measured peak is captured at the corresponding predicted time, thresholds can be employed. These thresholds can define how much earlier the measured peak can be than the corresponding predicted time to be captured and still be accepted as a match for the predicted time. These thresholds can also define symmetric or asymmetric intervals around the corresponding predicted time (in both directions).

[0178] In some embodiments, according to the principles described above regarding Figure 8 , the execution of the process described above regarding Figure 12a can be limited to those pixels / histograms related to the identified region of interest in the scene. That is, only the histograms that are of interest in finding NLOS objects relevant to the task (e.g., determining whether the parking space is empty) are evaluated. In this way, only those histograms that are more likely to contain strong secondary or subsequent peaks are evaluated.

[0179] Figure 12b An example of the predicted phasor and the measured iToF phasor diagram in the case where the parking space is not empty is schematically shown.

[0180] Since the iToF camera can only measure one phase, the measured phase Φ is the phase of the phasor 1214 derived by vector addition of the phasors 1211 and 1212 of the reflected signals of the primary and secondary reflections.

[0181] Depicts the measured Figure 10 in the case where the ToF camera is an indirect ToF camera. Figure 12a The prediction of the arrival time and phases 921 and 922 in Figure 8 is based on the assumption that the parking space is empty and the light beam (optical path l2 in Figure 9b ) can freely traverse the empty parking space. Therefore, the predicted phase is the same as in

[0182] Figure 12b The measured phasor 1214 is also shown inFigure 10 derived by vector addition of the phasors 1211 and 1212 of the primary and secondary reflections in the case shown.

[0183] The measured phasor 1214 deviates only slightly from the predicted phase 921(l1). This is because the distance traveled by the diffusely reflected light beam 107 at the vehicle 104 is shorter than the predicted optical path (see Figure 8 l1 and l2 in). Therefore, the time delay along the optical path ( Figure 10 106 and 107 in) is not as large as in the case where the light beam can traverse the optical path ( Figure 8 l2 in). The smaller the time delay, the smaller the measured phase.

[0184] This also means that the phasor 1212 is not azimuthally aligned with the predicted phase 922, which is predicted under the assumption that the parking space is empty.

[0185] The phasor 1215 corresponds to the phasor 1212, but its starting point is located at the end point of the phasor 1211. The starting point of the phasor 1211 and the end point of the phasor 1215 correspond to the starting point and the end point of the measured phasor 1214, representing the vector addition of the phasors 1211 and 1212.

[0186] If the parking space is empty as explained in the reference Figure 9b , the phase Φ of the measured phasor 1214 must be assigned to the measurement signals of the primary and secondary reflections, or only the primary reflection is overlapped with the reflection on the object in the parking space, thus shortening the optical path to the secondary reflection.

[0187] In Figure 12b 's example, the measured phasor 1214 deviates only slightly from the predicted phase 921 of the primary reflection, so it can be assumed that the light beam cannot freely traverse the optical path ( Figure 6 107 in and Figure 8 l2 in), and the parking space is not empty. This deviation can be classified using a threshold or a hypothesis test, for example, as described in the reference Figure 19 .

[0188] In some embodiments, according to the principles described above with respect to Figure 8 , the execution of the process described above with respect to Figure 12b can be limited to those pixels / phased maps of the iToF camera related to the identified region of interest in the scene. That is, only those histograms that are of interest for finding task-related NLOS objects (e.g., determining whether the parking space is empty) are evaluated. In this way, only those phased maps that are more likely to contain strong secondary or subsequent peaks can be evaluated.

[0189] Figure 13aA flowchart schematically showing the process of determining the status of a parking space in the case of using a dToF camera.

[0190] At S1, a vehicle model is determined based on the 3D model of the scene. This determination of the vehicle model can be achieved, for example, by a process as described regarding Figure 7a and Figure 7b At S2, the arrival time of photons with multipath reflection is predicted based on the vehicle model. This prediction of the arrival time of photons with multipath reflection can be achieved, for example, by a process as described regarding Figure 8 At S3, the position of the peak related to multipath reflection in the photon histogram is identified. The identification of the position of the peak in the photon histogram can be achieved, for example, by a process as described regarding Figure 9a At S4, the status of the parking space (e.g., whether the parking space is occupied or empty) is determined based on the predicted arrival time and the position of the peak in the photon histogram. This determination of the status of the parking space can be achieved, for example, by a process as described regarding Figure 12a It should be noted that, as described above regarding

[0191] The point cloud of the scene obtained at S1 can be based on ToF measurements (e.g., based on depth images, photon histograms, etc.). Alternatively or additionally, obtaining the point cloud can be based on information from auxiliary sensors such as RGB images or monochromatic images. Figure 2 As described regarding

[0192] Figure 13b A flowchart schematically showing the process of determining the status of a parking space in the case of using an iToF camera.

[0193] At S1, a vehicle model is determined based on the 3D model of the scene. This determination of the vehicle model can be achieved, for example, by a process as described regarding Figure 7a and Figure 7b At S2, the arrival time of photons with multipath reflection is predicted based on the vehicle model. This prediction of the arrival time of photons with multipath reflection can be achieved, for example, by a process as described regarding Figure 8 At S3, the phasor related to multipath reflection in the phase diagram is identified. This identification of the phase of the phasor can be achieved, for example, by a process as described regarding Figure 9a At S4, the status of the parking space (e.g., whether the parking space is occupied or empty) is determined based on the predicted phase and the phase of the phasor in the phase diagram. This determination of the status of the parking space can be achieved, for example, by a process as described regarding Figure 12b It should be noted that, as described above regarding

[0194] As described above regarding Figure 2As described, the point cloud of the scene obtained at S1 can be based on ToF measurements (e.g., based on depth images, photon histograms, etc.). Alternatively or additionally, obtaining the point cloud can be based on information from auxiliary sensors such as RGB images or monochrome images.

[0195] Example implementations

[0196] Below, some exemplary implementation aspects of the present disclosure are described.

[0197] Photon counting ToF system

[0198] A time-of-flight (ToF) camera is a range imaging camera system that determines the distance to an object by measuring the time of flight of an optical signal between the camera and the object for each point of the image. Typically, a ToF camera has an illumination unit (LED or VCSEL, vertical cavity surface emitting laser, EEL edge emitting laser) that illuminates the scene with modulated light. The pixel array in the ToF camera collects the light reflected from the scene. The indirect ToF (iToF) principle measures the phase shift between the transmitted signal and the received signal, which provides information about the light travel time and thus about the distance.

[0199] The direct ToF (dToF) method uses a photon counting (PC) technique pulse method, where the pulse width of the laser pulses generated by the lidar system can be varied. By reducing the pulse width, reflections are more easily distinguishable and higher resolution is achieved.

[0200] Photon counting ToF systems (such as dToF systems) are recording photon histograms, such as those described above with respect to FIGS. 9 and 12. Current systems are considered to use single photon avalanche diodes (SPADs) as detectors.

[0201] The binning of the histogram represents the measured distance, i.e., the distance of the object determined according to the ToF principle. Each bin is assigned the number of detections ("counts") in the corresponding bin (i.e., the corresponding spatial distance). The structure of the distribution (i.e., the "photon histogram" obtained in this way) depends on the actual distance of the detected object, the object itself, and the angle of the object. Further dependencies of the histogram are scattered rays or echoes that are not directly in the line of sight and may introduce noise into the measurement signal.

[0202] PC-ToF uses the number of photons falling into two or more consecutive bins during depth calculation to obtain sub-bin resolution. In contrast to dToF, PC-TOF systems use a relatively small number of bins. The duration of one bin is the same as the light pulse duration and can be greater than the duration of dToF.

[0203] A photon counting ToF system (PC-TOF) illuminates a scene with laser pulses, and the photons of the reflected light captured by the ToF sensor are evaluated in a photon histogram. The recording period for capturing the histogram is segmented into recording time slots of generally equal length. Photons arriving in the same recording time slot (spanning multiple recording periods) are attributed to the corresponding "bins" of the photon histogram. Photon counts from multiple such recording periods are typically aggregated into a single photon histogram. There may be a variable delay between consecutive recording periods.

[0204] Figure 14a The operating principle of a SPAD photodiode is schematically shown. A SPAD is an avalanche photodiode (APD) that operates in a voltage range V that exceeds the negative breakdown voltage V BD within the voltage range V (indicated by the SPAD in Figure 12a ), an electron-hole pair generated by a single photon triggers an avalanche effect, as indicated by arrow 1205. The avalanche effect causes a macroscopic current I to flow through the diode. The SPAD photodiode can be used in a dToF sensor.

[0205] It is expected that SPAD-based image sensors will allow gigapixel resolution, possibly using NAPD (nano-multiplication-region avalanche photodiode) or variants of this concept, such as those described by Kang L. Wang et al. in "Towards Ultimate Single Photon Counting Imaging CMOS Applications" at the Symposium on Astronomy and Space Science on January 5th and 6th, 2011, or by Xinyu Zheng et al. in "Modeling and Fabrication of a Nano-multiplication-region Avalanche Photodiode" on esto.nasa.gov in January 2007. Generally, the purpose of such a high pixel count is to achieve the so-called "digital movie", where the spatial count of photons arriving within the sensor area provides the final pixel value. If photons are counted in time rather than in space, the high pixel count of the image sensor can be used to increase the capture modality. This means that multi-spectral / hyperspectral acquisition, polarization, and even ToF functionality can be integrated into the same sensor.

[0206] SPAD pixels are binary devices: their output is 0 (no photons arrive) or 1 (photons arrive). Therefore, SPAD pixels cannot measure continuous intensity values. To achieve non-binary pixel values using a SPAD sensor, "photon counting" can be applied, i.e., counting the photons impinging on a pixel (or a region of the sensor) over time.

[0207] Figure 15 An example of the process for determining the pixel value according to the "photon counting" method is schematically provided. At 1401, photons generate electron-hole pairs on the SPAD pixel and trigger the corresponding avalanche effect, causing a macroscopic current to flow through the diode corresponding to the photon arrival according to Poisson statistics. At 1402, the number of avalanches generated by the SPAD pixel within a predetermined time interval Dt is counted to determine the corresponding number of arrivals in the time interval Dt. At 1403, the average number of arrivals is determined based on the number of avalanches generated by the SPAD pixel within a predetermined time interval Dt. At 1404, the pixel value is obtained according to the average number of arrivals obtained at 1403.

[0208] As an alternative to "photon counting", a previously proposed solution uses a "spatial counting" method: the number of SPAD pixels with states 0 and 1 is counted within a given area of the sensor, and these values 0 and 1 are converted into the intensity of the area under consideration. This method reduces the effective resolution because multiple SPAD pixels are used to simulate a single conventional pixel. That is, the SPAD sensor allows very high-resolution binary imaging, but in order to simulate the continuous intensity of a conventional pixel, spatial counting sacrifices this high resolution, thus limiting the benefits provided by the SPAD pixel itself.

[0209] Figure 14b The basic operating principle of an indirect time-of-flight imaging system that can be used for depth sensing is schematically shown. The iToF imaging system 1411 includes an iToF camera having an imaging sensor 1412 and a processor (CPU) 1415, and the imaging sensor has a pixel matrix. The scene 1417 is actively illuminated using an illumination device 1419 with amplitude-modulated infrared light LMS at a predetermined wavelength (e.g., some light pulses with at least one predetermined modulation frequency DML generated by a timing generator 1416). The amplitude-modulated infrared light LMS is reflected from an object within the scene 1417. The lens 1413 collects the reflected light 1419 and forms an image of the object within the scene 1417 onto the imaging sensor 1412. In indirect time-of-flight (iToF), the CPU 1415 determines the phase delay between the modulation signal DML and the reflected light RL for each pixel. Based on these correlations, the so-called in-phase component value ("I value") and the so-called quadrature component value ("Q value") can be determined for each pixel (for a detailed description, see below).

[0210] Figure 14bThe principle of the time-of-flight imaging system is described by taking the indirect time-of-flight imaging system as an example. However, the embodiments described below are not limited to the indirect time-of-flight principle. The depth-sensing pixels can also be, for example, iToF pixels (CAPD, gated ToF, etc.) or dToF pixels (SPAD) or PC pixels (SPAD) or dynamic photodiodes (DPD), etc. That is, the ToF pixels can also be implemented according to the dToF (direct TOF), iToF (indirect ToF), or PC (photon counting) principle.

[0211] KinectFusion

[0212] Figure 16 An example of implementing 3D reconstruction is shown. This example follows the method proposed by R.A. Newcombe et al. in "KinectFusion: Real-time dense surface mapping and tracking" (Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, 2011, pp. 127 - 136) (hereinafter also referred to as the "KinectFusion" method). KinectFusion describes a technique in which a real-time depth map stream is received and real-time dense SLAM is performed to gradually generate a consistent 3D scene model while tracking the agile movement of the ToF camera using all the depth data in each frame.

[0213] The surface measurement 1501 of the ToF data path receives the depth map D Figure 2 (u) of each pixel of the current frame k of the scene ( k (u) in 201) from the ToF camera to obtain a point cloud represented as a vertex map V k,c and a normal map N k,c . The subscript "c" represents camera coordinates.

[0214] The pose estimation 1502 of the 3D reconstruction is based on the point cloud V k,c , N k,c and the model feedback T g,k-1 to estimate the pose T g,k of the sensor. The subscript "g" represents global coordinates.

[0215] The model reconstruction 1503 of the 3D reconstruction performs surface reconstruction updates based on the estimated pose T k and the depth measurement D k (u) and provides an updated 3D model S Figure 2 of the scene ( k ) in 201.

[0216] The surface prediction 1504 receives the updated model Sk and determine from the currently estimated pose T g,k the dense 3D model surface prediction of the observed scene ( Figure 2 in 201) which yields a model estimated vertex map represented in the ToF camera coordinate system of the current frame k and the model estimated normal vectors

[0217] Surface measurement

[0218] The surface measurement 1501 of the ToF data path receives the depth map D Figure 2 in 201) of each pixel of the current frame k of the scene from the ToF camera k (u) to obtain the vertex map V k,c and the normal map N k,c of the point cloud. Each pixel is characterized by its corresponding (2D) image domain coordinates u=(u1,u2), where the depth measurement D k (u) of each pixel u of the current frame k is combined to yield the depth map D k of the current frame k. This yields the vertex map V k,c (u) (i.e., the metric point measurement in the ToF sensor coordinate system of the current frame k), also referred to as the point cloud V k,c . Prior to the transformation, a bilateral filter or any other noise reduction filter known in the art (anisotropic diffusion, non-local means, etc.) can be applied to the depth measurement D k (u). Additionally, the measurement 1501 also determines the normal vector N k,c (u) of each pixel u in the ToF camera coordinate system.

[0219] Using the camera calibration matrix K which includes the intrinsic camera configuration parameters, each pixel u in the image domain coordinates and its corresponding depth measurement D k (u) are transformed into the three-dimensional vertices corresponding to the current frame K within the ToF camera coordinate system

[0220]

[0221] This transformation is applied to each pixel u of the current frame k and its corresponding depth measurement D k (u), which yields the vertex map V k,c (u) (i.e., the metric point measurement in the ToF sensor coordinate system of the current frame k), also referred to as the point cloud V k,c . Additionally, the measurement 1501 also determines the normal vector N k,c (u) of each pixel u in the ToF camera coordinate system.

[0222] Pose estimation

[0223] The pose estimation of 3D reconstruction 1502 receives the vertex map V of each pixel u in the camera coordinate system corresponding to the current frame k k,c (u) and the normal vector N k,c (u), and receives the vertex map from surface prediction 1504 (see below), along with the latest available model updated based on the previous frame k - 1 of the model estimation and the normal vector of each pixel u of the model estimation. In another embodiment, the pose estimation can be directly based on the model S k , from which all points and all normals can be received through resampling. In addition, the pose estimation 1502 obtains the estimated pose T of the last frame k - 1 from the storage device g,k-1 . In another embodiment, more than one past pose can be used. For example, in a SLAM pipeline, there can be a separate (or "backend") thread that can perform online bundle adjustment and / or pose graph optimization to utilize all past poses.

[0224] Then, the pose estimation estimates the pose T of the current frame k g,k . The pose of the ToF camera describes the position and orientation of the ToF system, which is described by six degrees of freedom (6DOF), namely three position DOFs and three orientation DOFs. The three position DOFs are front / back, up / down, left / right, and the three orientation DOFs are yaw, pitch, and roll. The current pose of the ToF camera at frame k can be represented by a rigid body transformation, which is defined by the pose matrix T g,k :

[0225]

[0226] where, is the matrix representing the rotation of the ToF camera, and is the vector representing the translation of the ToF camera from the origin, where they are represented in the global coordinate system. SE(3) represents the so-called special Euclidean group in three dimensions. The pose estimation is based on the vertex map V of each pixel u of the current frame k k,c (u) and the normal vector N k,c (u), and the model estimation of the vertex map V k-1,c (u) and the model estimation of the normal vector N of each pixel u k,c (u) based on the latest available model updated to the previous frame k - 1. In another embodiment, the model S k is directly used, especially if it is a mesh model, such as by resampling the mesh. Further, it is based on the estimated pose T of the last frame k - 1 g,k-1 . The pose estimation estimates the pose T of the current frame k based on the iterative closest point (ICP) algorithmg,k , as explained in the aforementioned "KinectFusion" paper. According to the estimated pose T of the current frame k g,k , the vertex map V of the current frame k k,c (u) can be transformed into the global coordinate system, which results in the global vertex map V k,g (u):

[0227] V k,g (u) = R k ·V k,c (u) + t k (Equation 4)

[0228] When this operation is performed for all pixels u, it results in the registered point cloud V k,g . Therefore, the normal vector N of each pixel u of the current frame k k,c (u) can be transformed into the global coordinate system:

[0229] N k,g (u) = R k ·N k,c (u) (Equation 5)

[0230] Model reconstruction (surface reconstruction update)

[0231] Scene ( Figure 2 in 201) of the 3D model can be reconstructed, for example, based on the Truncated Signed Distance Function (TSDF) of the volume or other models as described below. The volume surface representation based on TSDF represents the 3D scene ( Figure 2 in 201) within the volume Vol as a voxel grid, where the TSDF model stores the signed distance to the nearest surface for each voxel p. The volume Vol is represented by an isometric voxel grid characterized by its center . The voxel p (i.e., its center) is given in the global coordinate system. The value of the TSDF at the voxel p corresponds to the signed distance to the nearest zero crossing (i.e., Figure 2 the surface interface of scene 201 in) and takes positive and increasing values when moving from the visible surface of the scene ( Figure 2 in 201) to free space, and takes negative and decreasing values on the invisible side of scene 201, where the function is truncated when the distance from the surface exceeds a certain distance. The result of iteratively fusing (averaging) the TSDFs of multiple 3D registered point clouds (from multiple frames) of the same scene 201 into the global 3D model produces the global TSDF model S k , which contains the fusion of frames 1,..., k of scene 201. The global TSDF model S k is described by two values for each voxel p within the volume Vol, namely the actual TSDF function F k (p) that describes the distance to the nearest surface and the evaluation of Fk Uncertainty weight W of the uncertainty of (p) k (p), i.e., S k := [F k (p), W k (p)]. Iteratively construct the global TSDF model S of scene 201 k , and incorporate the depth map D of scene 201 k and the corresponding pose estimate T of the current frame k g,k and fuse them into the previous global TSDF model S of scene 201 k-1 such that the updated and thus improved global TSDF model S is updated by the registered point cloud V of the current frame k k,g := [F k-1 (p), W k-1 (p)]. Therefore, the model reconstruction receives the depth map D of the current frame k k-1 and the current estimated pose T k (which generates the registered point cloud V of the current frame k k ), and outputs the updated global TSDF model S k,g := [F k (p), W k (p)]. This means that the updated global TSDF model S k := [F k (p), W k (p)] is based on the previous global TSDF model S k := [F k-1 (p), W k-1 (p)] and the current registered point cloud V k-1 . According to the "KinectFusion" paper cited above, it is determined that: k,g f(p, T

[0232] , D g,k (= Ψ * λ k ||t -1 - p||2 - D k (x)), (Equation 6), where: k λ = ||K

[0233] [x, 1] -1 ||2 (Equation 7) T ||2 (Equation 7)

[0234]

[0235] W k (p) = W k-1 (p) + w(p, N k,c ) (Equation 11)

[0236] w(p,N k,c ) ∝ cos(θ) / D k (x)

[0237] where the function y = π(z) performs a perspective projection including de - homogenization to obtain and where θ is the angle between the associated pixel ray direction and the surface normal measurement N k,c The TSDF is also described in more detail, for example, in the above - mentioned KinectFusion paper. Further, the model reconstruction 1503 can receive model feedback (e.g., model feedback matrix A k , see below), which indicates for each pixel whether it is reliable (overlapping pixels in the case of sufficient overlap and non - overlapping pixels in the case of insufficient overlap), unreliable (non - overlapping pixels in the case of sufficient overlap), or new (non - overlapping pixels in the case of insufficient overlap). The depth data of reliable or new pixels can be used to improve the model as described above (which means the model is created or updated with the corresponding depth measurements), and the depth data of unreliable pixels can be discarded or stored in a dedicated buffer that may or may not be used.

[0238] Vehicle control system

[0239] When implementing a ToF camera system in a vehicle, it is beneficial to incorporate the ToF camera system into the standard vehicle control system. This enables sharing of computer resources and hardware components among multiple systems.

[0240] The techniques according to embodiments of the present disclosure can be applied to various products. For example, the techniques of the embodiments can be used in a driving assistance system. For example, the techniques according to embodiments of the present disclosure can be implemented as a device included in a mobile body, which is any one of an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility vehicle, an aircraft, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.

[0241] Figure 17 is a block diagram showing an example of a schematic configuration of a vehicle control system 7000 that depicts an example of a mobile body control system to which the techniques according to embodiments of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. In Figure 17In the example depicted, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside vehicle information detection unit 7400, an inside vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 that connects the multiple control units to each other can be, for example, an in-vehicle communication network that complies with any standard, such as Controller Area Network (CAN), Local Interconnect Network (LIN), Local Area Network (LAN), FlexRay (registered trademark), etc.

[0242] Each control unit includes: a microcomputer that performs arithmetic processing according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, etc.; and a drive circuit that drives various control target devices. Each control unit also includes: a network interface (I / F) for communicating with other control units via the communication network 7010; and a communication I / F for communicating with devices, sensors, etc. inside and outside the vehicle through wired communication or radio communication. Figure 17 The functional configuration of the integrated control unit 7600 shown in includes a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon reception unit 7650, an in-vehicle device I / F 7660, a sound / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units similarly include a microcomputer, a communication I / F, a storage unit, etc.

[0243] The drive system control unit 7100 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 7100 serves as a control device for a driving force generation device (such as an internal combustion engine, a drive motor, etc.) that generates the driving force of the vehicle, a driving force transmission mechanism that transmits the driving force to the wheels, a steering mechanism that adjusts the steering angle of the vehicle, a braking device that generates the braking force of the vehicle, etc. The drive system control unit 7100 can have the function of a control device for an anti-lock braking system (ABS), an electronic stability control (ESC), etc.

[0244] The drive system control unit 7100 is connected to the vehicle state detection unit 7110. For example, the vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational movement of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, and sensors for detecting the operation amount of the accelerator pedal, the operation amount of the brake pedal, the steering angle of the steering wheel, the engine speed, or the rotational speed of the wheels. The drive system control unit 7100 performs arithmetic processing using the signals input from the vehicle state detection unit 7110 and controls the internal combustion engine, the drive motor, the electric power steering device, the braking device, and the like. Both the drive system control unit 7100 and the vehicle state detection unit 7110 can control the operation of the devices related to the vehicle's driving system based on the information measured by the ToF camera system.

[0245] The body system control unit 7200 controls the operation of various devices provided to the vehicle body according to various programs. For example, the body system control unit 7200 serves as a control device for a keyless entry system, a smart key system, an electric window device, or various lights such as headlights, reverse lights, brake lights, turn signal lights, fog lights, etc. In this case, radio waves or signals from a mobile device that substitutes for a key or signals from various switches can be input to the body system control unit 7200. The body system control unit 7200 receives these input radio waves or signals and controls the vehicle's door lock device, electric window device, lights, and the like.

[0246] The battery control unit 7300 controls the secondary battery 7310 that is the power source of the drive motor according to various programs. For example, information about the battery temperature, the battery output voltage, the remaining charge in the battery, etc. is provided from the battery device including the secondary battery 7310 to the battery control unit 7300. The battery control unit 7300 performs arithmetic processing using these signals and performs control for adjusting the temperature of the secondary battery 7310 or controls the cooling device provided to the battery device, etc.

[0247] The outboard information detection unit 7400 detects information about the outside of the vehicle including the vehicle control system 7000. For example, the outboard information detection unit 7400 is connected to at least one of the imaging unit 7410 and the outboard information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outboard information detection unit 7420 includes, for example, at least one of an environmental sensor for detecting the current atmospheric conditions or weather conditions and a peripheral information detection sensor for detecting another vehicle, an obstacle, a pedestrian, etc. on the periphery of the vehicle including the vehicle control system 7000. The outboard information detection unit 7400 including the imaging unit 7410 can include, for example, the ToF camera system and data path as described above regarding Figure 2 and the data path.

[0248] For example, the environmental sensor may be at least one of a raindrop sensor that detects rainfall, a fog sensor that detects fog, a sunlight sensor that detects the degree of sunlight, and a snow sensor that detects snowfall. The peripheral information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR device (light detection and ranging device, or laser imaging detection and ranging device). Each of the imaging unit 7410 and the out-vehicle information detection unit 7420 may be provided as an independent sensor or device, or may be provided as a device in which a plurality of sensors or devices are integrated.

[0249] Figure 18 An example of the installation positions of the imaging unit 7410 and the out-vehicle information detection unit 7420 is depicted. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least at one of positions on the front nose, side mirrors, rear bumper, and rear door of the vehicle 7900 and at a position on the upper portion of the windshield inside the vehicle. The imaging unit 7910 provided to the front nose inside the vehicle and the imaging unit 7918 provided to the upper portion of the windshield mainly obtain images of the front part of the vehicle 7900. The imaging units 7912 and 7914 provided to the side mirrors mainly obtain images of the sides of the vehicle 7900. The imaging unit 7916 provided to the rear bumper or rear door mainly obtains images of the rear part of the vehicle 7900. The imaging unit 7918 provided to the upper portion of the windshield inside the vehicle is mainly used to detect preceding vehicles, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0250] Incidentally, Figure 18 An example of the imaging ranges of the corresponding imaging units 7910, 7912, 7914, and 7916 is depicted. The imaging range a represents the imaging range of the imaging unit 7910 provided to the front nose. The imaging ranges b and c represent the imaging ranges of the imaging units 7914 and 7912 provided to the side mirrors, respectively. The imaging range d represents the imaging range of the imaging unit 7916 provided to the rear bumper or rear door. For example, by superimposing the image data imaged by the imaging units 7910, 7912, 7914, and 7916, an aerial view image of the vehicle 7900 viewed from above can be obtained.

[0251] The out-of-vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided for the front, rear, sides, and corners of the vehicle 7900 and the upper part of the windshield inside the vehicle can be, for example, ultrasonic sensors or radar devices. The out-of-vehicle information detection units 7920, 7926, and 7930 provided for the front nose of the vehicle 7900, the rear bumper, the rear door of the vehicle 7900, and the upper part of the windshield inside the vehicle can be, for example, LIDAR devices or ToF camera systems. These out-of-vehicle information detection units 7920 to 7930 are mainly used to detect vehicles ahead, pedestrians, obstacles, etc.

[0252] Return to Figure 17 , the description will continue. The out-of-vehicle information detection unit 7400 causes the imaging unit 7410 to image an image of the outside of the vehicle and receives the imaged image data. In addition, the out-of-vehicle information detection unit 7400 receives detection information from the out-of-vehicle information detection unit 7420 connected to the out-of-vehicle information detection unit 7400. When the out-of-vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, a LIDAR device, or a ToF camera system, the out-of-vehicle information detection unit 7400 transmits ultrasonic waves, radar waves, etc. and receives information on the received reflected waves. Based on the received information, the out-of-vehicle information detection unit 7400 can perform processing to detect objects such as people, vehicles, obstacles, signs, characters on the road surface, etc., or processing to detect the distance to the object. The out-of-vehicle information detection unit 7400 can perform environmental recognition processing to identify rainfall, fog, road surface conditions, etc. based on the received information. The out-of-vehicle information detection unit 7400 can calculate the distance to an out-of-vehicle object based on the received information.

[0253] In addition, based on the received image data, the out-of-vehicle information detection unit 7400 can perform image recognition processing to identify people, vehicles, obstacles, signs, characters on the road surface, etc., or processing to detect the distance to the object. The out-of-vehicle information detection unit 7400 can perform processing such as distortion correction and alignment on the received image data and combine the image data imaged by multiple different imaging units 7410 to generate a bird's-eye view image or a panoramic image. The out-of-vehicle information detection unit 7400 can perform viewpoint conversion processing using the image data imaged by the imaging unit 7410 including different imaging units.

[0254] The in-vehicle information detection unit 7500 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 7500 is connected to a driver state detection unit 7510 that detects the state of the driver. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the biometric information of the driver, a microphone that collects the sound inside the vehicle, and the like. The biosensor is provided, for example, on the seat surface, the steering wheel, etc., and detects the biometric information of the occupant sitting on the seat or the driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the in-vehicle information detection unit 7500 can calculate the degree of fatigue or the degree of concentration of the driver, or can determine whether the driver is dozing off. The in-vehicle information detection unit 7500 can process the audio signal obtained by collecting sound, such as noise cancellation processing and the like.

[0255] The integrated control unit 7600 controls the general operations within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to an input unit 7800. The input unit 7800 is implemented by a device that can be operated by the occupant. For example, a touch panel, a button, a microphone, a switch, a lever, etc. Data obtained by performing speech recognition on the speech input through the microphone can be provided to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device that uses infrared rays or other radio waves, or an external connection device that supports the operation of the vehicle control system 7000, such as a mobile phone, a personal digital assistant (PDA), etc. The input unit 7800 may be a camera, for example. In this case, the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of the wearable device worn by the occupant can be input. In addition, the input unit 7800 may include, for example, an input control circuit, etc., which generates an input signal based on the information input by the occupant or the like using the above input unit 7800, and outputs the generated input signal to the integrated control unit 7600. The occupant or the like inputs various data or gives instructions for processing operations to the vehicle control system 7000 by operating the input unit 7800.

[0256] The storage unit 7690 may include a read-only memory (ROM) that stores various programs executed by the microcomputer and a random access memory (RAM) that stores various parameters, operation results, sensor values, etc. In addition, the storage unit 7690 can be implemented by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc.

[0257] The general communication I / F 7620 is a widely used communication I / F that mediates communication with various devices existing in the external environment 7750. The general communication I / F 7620 can implement cellular communication protocols such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), Advanced LTE (LTE-A), etc., or another wireless communication protocol such as Wireless LAN (also known as Wi-Fi (registered trademark)), Bluetooth (registered trademark), etc. The general communication I / F 7620 can be connected, for example, via a base station or an access point, to a device (such as an application server or a control server) existing on an external network (such as the Internet, a cloud network, or a company private network). Additionally, the general communication I / F 7620 can use, for example, peer-to-peer (P2P) technology to connect to a terminal existing near the vehicle (the terminal is, for example, a driver's, a pedestrian's, or a store's terminal, or a machine type communication (MTC) terminal).

[0258] The dedicated communication I / F 7630 is a communication I / F that supports communication protocols developed for use in a vehicle. The dedicated communication I / F 7630 can implement standard protocols such as Wireless Access in Vehicular Environment (WAVE) in a vehicle environment, which is a combination of the Institute of Electrical and Electronics Engineers (IEEE) 802.11p as the lower layer and IEEE 1609 as the upper layer, Dedicated Short Range Communications (DSRC), or a cellular communication protocol. The dedicated communication I / F 7630 generally performs V2X communication as a concept that includes one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0259] For example, the positioning unit 7640 performs positioning by receiving Global Navigation Satellite System (GNSS) signals (such as GPS signals from GPS satellites) from GNSS satellites, and generates position information including the latitude, longitude, and altitude of the vehicle. Incidentally, the positioning unit 7640 can identify the current position by exchanging signals with a wireless access point, or can obtain position information from a terminal such as a mobile phone, a Personal Handy-phone System (PHS), or a smart phone with a positioning function.

[0260] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from a wireless station installed on a road or the like, and thereby obtains information about the current position, congestion, closed roads, necessary time, etc. Incidentally, the function of the beacon receiving unit 7650 can be included in the above-mentioned dedicated communication I / F 7630.

[0261] The in-vehicle device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various in-vehicle devices 7760 existing in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using a wireless communication protocol such as Wireless LAN, Bluetooth (registered trademark), Near Field Communication (NFC), or Wireless Universal Serial Bus (WUSB). Additionally, the in-vehicle device I / F 7660 can establish a wired connection via a connection terminal (and a cable if necessary) not depicted in the figure through Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI (registered trademark)), Mobile High-Definition Link (MHL), etc. The in-vehicle device 7760 can include, for example, at least one of a mobile device and a wearable device owned by a passenger and an information device carried into the vehicle or attached to the vehicle. The in-vehicle device 7760 can also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0262] The vehicle-mounted network I / F 7680 is an interface that mediates the communication between the microcomputer 7610 and the communication network 7010. The vehicle-mounted network I / F 7680 transmits and receives signals, etc., according to a predetermined protocol supported by the communication network 7010.

[0263] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon reception unit 7650, the in-vehicle device I / F 7660, and the vehicle-mounted network I / F 7680. For example, the microcomputer 7610 can calculate control target values for a driving force generation device, a steering mechanism, or a braking device based on the obtained information about the inside and outside of the vehicle, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 can execute cooperative control for implementing functions of an Advanced Driver Assistance System (ADAS), which includes collision avoidance or shock absorption for the vehicle, following driving based on a following distance, vehicle speed-holding driving, vehicle collision warning, vehicle lane departure warning, etc. Additionally, the microcomputer 7610 can execute cooperative control for autonomous driving by controlling a driving force generation device, a steering mechanism, a braking device, etc., based on the obtained information about the surroundings of the vehicle, so that the vehicle can autonomously travel without depending on the operation of the driver, etc.

[0264] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680, and generate local map information including information about the surroundings of the vehicle's current position. The three-dimensional distance information can also be generated based on the measurement of the ToF camera system. Additionally, the microcomputer 7610 can predict risks such as vehicle collisions, pedestrian approaches, and entry into a closed road based on the obtained information, and generate a warning signal. For example, the warning signal can be a signal for generating a warning sound or lighting a warning lamp.

[0265] The microcomputer 7610 can, for example, execute the processes described in the above-described embodiments, such as the steps described in FIG. 13. This can be done based on the measurement data of the ToF camera system, which is an example of the out-vehicle information detection unit 7400 including the imaging unit 7410.

[0266] The sound / image output unit 7670 sends an output signal of at least one of sound and image to an output device capable of notifying information visually or auditorily to the occupants of the vehicle or outside the vehicle. In Figure 17 the example, the audio speaker 7710, the display unit 7720, and the instrument panel 7730 are illustrated as output devices. The display unit 7720 can include, for example, at least one of an on-board display and a head-up display. The display unit 7720 can have an augmented reality (AR) display function. The output device can be other devices than these devices, and can be another device such as a headphone, a wearable device such as a glasses-type display worn by the occupant, a projector, a lamp, etc. When the output device is a display device, the display device visually displays the results obtained by various processes executed by the microcomputer 7610 or the information received from another control unit in various forms such as text, image, table, chart, etc. Additionally, when the output device is an audio output device, the audio output device converts an audio signal composed of reproduced audio data or sound data, etc. into an analog signal and outputs the analog signal auditorily.

[0267] Incidentally, in Figure 17In the example depicted in, at least two control units connected to each other via the communication network 7010 can be integrated into one control unit. Alternatively, each individual control unit may include multiple control units. In addition, the vehicle control system 7000 may include another control unit not depicted in the figure. In addition, part or all of the functions performed by a control unit in the above description can be assigned to another control unit. That is, as long as information is sent and received via the communication network 7010, predetermined arithmetic processing can be performed by any control unit. Similarly, a sensor or device connected to one control unit can be connected to another control unit, and multiple control units can send and receive detection information to each other via the communication network 7010.

[0268] Incidentally, a computer program for realizing the function of parking space detection may be implemented in one of the control unit and the like. In addition, a computer-readable recording medium storing such a computer program may also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above-mentioned computer program may be distributed, for example, via a network without using a recording medium.

[0269] In the vehicle control system 7000, the parking space detection function according to the above embodiment can be applied to Figure 17 The integrated control unit 7600 in the application example shown in .

[0270] Figure 19 A diagram schematically shows a binary Bayesian hypothesis test for classifying measurement results in the case of iToF measurements.

[0271] The iToF phase diagram is depicted (e.g., Figure 9b or Figure 12b ) in the probability density functions 1901 and 1902, the phase 1903 as the center value of the probability density function 1901, the phase 1905 as the center value probability density function 1902, and the phase 1904 of the measured phasor.

[0272] The phase increases from left to right. Therefore, probability density function 1901 is the probability density function of the hypothesis H0 that the parking space is not empty (occupied), which means that the measured iToF signal does not include the reference Figure 12b The secondary reflections described, or the secondary reflections, have minimal influence on the actual measurement of the phase, which means that the optical path ( Figure 8 The phase of l1+l2) in the optical path is very close to that of Figure 8 The probability density function 1901 indicates that the measured light has traveled the optical path of the primary reflection ( Figure 8 Due to the earlier arrival time, a smaller phase is expected in the case of a non-empty parking space.

[0273] The probability density function 1902 is the probability density function of the hypothesis H1 that the parking space is empty, which means that the measured iToF signal indeed includes a phase generated by the vector product of the phasors of the primary reflection and the secondary reflection as described in the reference Figure 12b The probability density function 1902 represents the hypothesis that the measured light has traveled the optical paths of the primary reflection and the secondary reflection ( Figure 8 l1 and l2 in ).

[0274] The probability density functions 1901 and 1902 can be predetermined by the system or modified consistently. For example, they originate from the measurement uncertainty of the system, which propagates into the creation of the model and thus results in the uncertainty of the position of the reflecting surface, and thus the probability density functions of the length of the traversed optical path and the measured time delay or phase.

[0275] One way to select which hypotheses (probability density functions) the measured phase 1904 most likely belongs to is the maximum a posteriori test. Here, the posterior probability of the phase 1904 is compared as the value of any hypothesis, and the hypothesis with the larger posterior probability is selected.

[0276] If the hypothesis H0 is the hypothesis of the probability density function 1901, the hypothesis H1 is the hypothesis of the probability density function 1902, and Φ m is the phase 1904, then the posterior probability of H0 is:

[0277]

[0278] And the posterior probability of H1 is:

[0279]

[0280] where, f Φ (Φ m |H0) is the probability that Φ m is measured when the hypothesis H0 is true, that is, the value of the probability density function 1901 at the phase 1904, P(H0) is the prior probability of H0, f Φ (Φ m ) is the probability that Φ m is measured, f Φ (Φ m |H1) is the probability that Φ m is measured when the hypothesis H1 is true, that is, the value of the probability density function 1902 at the phase 1904, and P(H1) is the prior probability of H1. Therefore, if the following condition is met, then H0 is selected:

[0281] P(H0|Φ=Φ m )≥P(H1|Φ=Φm ) (Equation 14)

[0282] Alternatively, for simplicity, if the following conditions are met, then select H0:

[0283] f Φ (Φ m |H0)P(H0) ≥ f Φ (Φ m |H1)P(H1) (Equation 15)

[0284] If H0 is not selected, then select H1. This classification can be used instead of the reference Figure 12a discussed threshold.

[0285] Phase 1904 is the phase of the phasor measured by an iToF system (e.g., Figure 9b or Figure 12b ) (the phasor in Figure 9b 914 and Figure 12b 1214 in). Therefore, the generally known binary Bayesian hypothesis test is used to classify which probability density function the phasor of phase 1904 most likely belongs to, and thus to classify whether the parking space is empty. Additionally, a minimum-cost hypothesis test can be employed. This classification can be used instead of the reference Figure 12b discussed threshold.

[0286] ***

[0287] It should be recognized that the embodiments describe a method with method steps having an exemplary ordering. However, the specific ordering of the method steps is provided for illustrative purposes only and should not be construed as binding.

[0288] It should be noted that the present disclosure is not limited to any specific functional partitioning in a particular unit.

[0289] The method can also be implemented as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor (such as the microcomputer 7610 discussed above) to execute the method. In some embodiments, a non-transitory computer-readable recording medium is also provided, in which a computer program product is stored that, when executed by a processor (such as the above-mentioned processor), causes the described method to be executed.

[0290] Unless otherwise specified, all units and entities described in this specification and claimed in the appended claims can be implemented as integrated circuit logic (e.g., on a chip), and unless otherwise specified, the functions provided by such units and entities can be implemented by software.

[0291] The processing system described above can be implemented, for example, by a corresponding programmed processor, a field programmable gate array (FPGA), or the like.

[0292] Insofar as the above-disclosed embodiments are implemented using at least partially software-controlled data processing devices, it will be understood that providing such a software-controlled computer program and providing a transmission, storage, or other medium for such a computer program are contemplated aspects of the present disclosure.

[0293] Note that the present technology can also be configured as described below.

[0294] (1) An electronic device (200) including circuitry configured to detect a non-line-of-sight object based on a comparison between ToF information obtained from reflected light received from a ToF imaging sensor (110) and model-based information obtained from a model-based prediction of light reflection.

[0295] (2) The electronic device (200) according to (1), wherein the circuitry is configured to detect at least one of the presence, position, and velocity of the non-line-of-sight object (110).

[0296] (3) The electronic device (200) according to (1) or (2), wherein the circuitry is configured to determine the state of a point based on the detection of the non-line-of-sight object.

[0297] (4) The electronic device (200) according to (3), wherein the state of the point includes information about whether the point is empty.

[0298] (5) The electronic device (200) according to (3) or (4), wherein the point is a parking space.

[0299] (6) The electronic device (200) according to any one of (1) to (5), wherein the ToF information includes information about the time of arrival of photons related to multipath reflection.

[0300] (7) The electronic device (200) according to any one of (1) to (6), wherein the circuitry is configured to obtain the model-based information through a ray tracing process.

[0301] (8) The electronic device (200) according to any one of (1) to (7), wherein the model-based information includes a predicted time of arrival of photons.

[0302] (9) The electronic device (200) according to (8), wherein the predicted time of arrival of photons is related to at least one of second-order light reflection, third-order light reflection, and higher-order light reflection.

[0303] The electronic device (200) according to any one of (1) to (9), wherein the model-based information includes a prediction of multipath light reflection.

[0304] (11) The electronic device (200) according to any one of (1) to (10), wherein the circuit is configured to determine whether the ToF information obtained from the reflected light deviates from the model-based information.

[0305] (12) The electronic device (200) according to any one of (1) to (11), wherein the circuit is configured to determine whether the position of the reflection indicated by the ToF information deviates from the model-based prediction.

[0306] (13) The electronic device (200) according to any one of (1) to (12), wherein the circuit is configured to perform model-based light reflection prediction based on a reconstructed 3D model of the captured scene (301).

[0307] (14) The electronic device (200) according to any one of (1) to (13), wherein the circuit is configured to perform model-based light reflection prediction based on one or more vehicle models (VM).

[0308] (15) The electronic device (200) according to (14), wherein the vehicle model (VM) models components of the vehicle (102, 103) that are not visible to the ToF imaging sensor (110) in the imaging information obtained from the primary reflection.

[0309] (16) The electronic device (200) according to (14) or (15), wherein the circuit is configured to determine the vehicle model (VM) based on a 3D model of the scene (301).

[0310] (17) The electronic device (200) according to (16), wherein the vehicle model (VM) models components of the vehicle (102, 103) that are not present in the 3D model of the scene (301).

[0311] (18) The electronic device (200) according to any one of (1) to (17), wherein the ToF imaging sensor (110) is a dToF imaging sensor.

[0312] (19) The electronic device (200) according to (18), wherein the circuit is configured to obtain the ToF information from a photon histogram captured by the ToF imaging sensor (110).

[0313] (20) The electronic device (200) according to any one of (1) to (17), wherein the ToF imaging sensor (110) is an iToF imaging sensor.

[0314] The electronic device (200) according to (20), wherein the circuit is configured to obtain ToF information from the phases captured by the ToF imaging sensor (110).

[0315] The electronic device (200) according to (20), wherein non-line-of-sight objects are detected based on the phases captured by ToF imaging and predicted phases.

[0316] The electronic device (200) according to (22), wherein the predicted phase is predicted based on model-based information obtained from model-based multipath light reflection prediction.

[0317] The electronic device (200) according to any one of (1) to (23), wherein the circuit is configured to emit light and obtain ToF information from at least one of second-order reflections, third-order reflections, and higher-order reflections of the emitted light.

[0318] The electronic device (200) according to any one of (1) to (24), wherein the circuit is configured to emit light, and wherein the circuit is configured to model the optical path of the emitted light to obtain model-based light reflection prediction.

[0319] A method, comprising: detecting non-line-of-sight objects based on a comparison between ToF information obtained from reflected light received by a ToF imaging sensor (110) and model-based information obtained from model-based light reflection prediction.

[0320] A computer program, comprising instructions that are configured to, when executed on a processor, perform the method according to (26).

Claims

1. An electronic device includes a circuit configured to detect non-line-of-sight objects based on a comparison between ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection.

2. The electronic device according to claim 1, wherein, The circuit is configured to obtain the ToF information from a photon histogram captured by the ToF imaging sensor.

3. The electronic device according to claim 1, wherein, The circuit is configured to detect at least one of the presence, position, and velocity of the non-line-of-sight object.

4. The electronic device according to claim 1, wherein, The circuit is configured to determine the state of a point based on the detection of the non-line-of-sight object.

5. The electronic device according to claim 4, wherein, The state of the point includes information about whether the point is empty.

6. The electronic device according to claim 4, wherein, The point is a parking space.

7. The electronic device according to claim 1, wherein, The ToF information includes information about the arrival time of photons related to multipath reflection.

8. The electronic device according to claim 1, wherein, The circuit is configured to obtain the model-based information through a ray tracing process.

9. The electronic device according to claim 1, wherein, The model-based information includes a predicted photon arrival time.

10. The electronic device according to claim 9, wherein, The predicted photon arrival time is related to at least one of second-order light reflection, third-order light reflection, and higher-order light reflection.

11. The electronic device according to claim 1, wherein, The model-based information includes a prediction of multipath light reflection.

12. The electronic device according to claim 1, wherein, The circuit is configured to determine whether the ToF information obtained from the reflected light deviates from the model-based information.

13. The electronic device according to claim 1, wherein, The circuit is configured to determine whether the position of the reflection indicated by the ToF information deviates from a model-based prediction.

14. The electronic device according to claim 1, wherein, The circuit is configured to perform the model-based prediction of light reflection based on a reconstructed 3D model of the captured scene.

15. The electronic device according to claim 1, wherein, The circuit is configured to perform the model-based prediction of light reflection based on one or more vehicle models.

16. The electronic device according to claim 15, wherein, The vehicle model models components of a vehicle that are not visible to the ToF imaging sensor in the imaging information obtained from the primary reflection.

17. The electronic device according to claim 15, wherein, The circuit is configured to determine the vehicle model based on a 3D model of the scene.

18. The electronic device according to claim 17, wherein, The vehicle model models components of a vehicle that do not exist in the 3D model of the scene.

19. The electronic device according to claim 1, wherein The circuit is configured to emit light and obtain the ToF information from at least one of second-order reflection, third-order reflection, and higher-order reflection of the emitted light.

20. The electronic device according to claim 1, wherein, The circuit is configured to emit light, and wherein the circuit is configured to model the optical path of the emitted light to obtain the model-based prediction of light reflection.

21. A method for detecting non-line-of-sight objects based on a comparison between ToF information obtained from reflected light received by a ToF imaging sensor and model-based information obtained from a model-based prediction of light reflection.

22. A computer program includes instructions configured to perform the method according to claim 21 when executed on a processor.

Images