Method and system for obstacle detection using resolution-adaptive fusion of point clouds

Abstract

Disclosed are methods and systems for obstacle detection using resolution-adaptive fusion of point clouds. A method for obstacle detection and vehicle navigation using resolution-adaptive fusion includes performing, by a processor, resolution-adaptive fusion of at least a first three-dimensional (3D) point cloud and a second 3D point cloud to produce a fused, noise-reduced, and resolution-optimized 3D point cloud representing an environment associated with a vehicle. The first 3D point cloud is generated by a first type of 3D scanning sensor, and the second 3D point cloud is generated by a second type of 3D scanning sensor. The second type of 3D scanning sensor has a different resolution in each of a plurality of different measurement dimensions relative to the first type of 3D scanning sensor. The method also includes detecting obstacles and navigating the vehicle using the fused, noise-reduced, and resolution-optimized 3D point cloud.

Description

Technical Field

[0001] The present disclosure relates to obstacle detection and vehicle navigation, and more particularly to obstacle detection and vehicle navigation using resolution-adaptive fusion of point clouds. Background Art

[0002] Autonomous vehicles may use three-dimensional (3D) scanning devices, such as light detection and ranging (LIDAR) sensors, vision sensors, radar, or other 3D sensing devices, for obstacle detection and vehicle navigation. Such sensors are capable of generating point clouds that can be combined using sensor fusion methods or processes. Some existing sensor fusion methods process the raw point cloud data from each sensor separately and independently, and then perform fusion by combining object trajectories from different sensors. Other sensor fusion methods perform fusion at the point cloud level, but only by registering and aligning the point clouds in a common coordinate system and then fitting a mesh or surface to the composite point cloud. These existing point cloud fusion methods do not take into account the resolution differences between sensors and cannot compensate for the poor resolution of a particular type of sensor in one or more dimensions. Therefore, there is a need to process point clouds from different types of sensors that take into account the sensor resolution and automatically compensate for the poor resolution of the sensors to detect obstacles for use in vehicle navigation. Summary of the Invention

[0003] According to an embodiment, a method for obstacle detection and vehicle navigation using resolution-adaptive fusion includes performing, by a processor, resolution-adaptive fusion of at least a first three-dimensional (3D) point cloud and a second 3D point cloud to generate a fused, noise-reduced, and resolution-optimized 3D point cloud representing an environment associated with the vehicle. The first 3D point cloud is generated by a first type of 3D scanning sensor and the second 3D point cloud is generated by a second type of 3D scanning sensor. The second type of 3D scanning sensor includes a different resolution in each of a plurality of different measurement dimensions relative to the first type of 3D scanning sensor. The method also includes detecting obstacles and navigating the vehicle using the fused, noise-reduced, and resolution-optimized 3D point cloud.

[0004] According to an embodiment, a system for obstacle detection and vehicle navigation using resolution adaptive fusion includes a processor and a memory associated with the processor. The memory includes computer-readable program instructions that, when executed by the processor, cause the processor to perform a set of functions. The set of functions includes: performing resolution adaptive fusion of at least a first three-dimensional (3D) point cloud and a second 3D point cloud to generate a fused, denoised, and resolution-optimized 3D point cloud representing an environment associated with the vehicle. The first 3D point cloud is generated by a first type of 3D scanning sensor, and the second 3D point cloud is generated by a second type of 3D scanning sensor. Relative to the first type of 3D scanning sensor, the second type of 3D scanning sensor has a different resolution in each of a plurality of different measurement dimensions. The set of functions also includes: using the fused, denoised, and resolution-optimized 3D point cloud to detect obstacles and navigate the vehicle.

[0005] According to an embodiment and any of the preceding embodiments, performing the resolution-adaptive fusion includes generating a first volume surface function by 3D convolving each of the plurality of measurement points of the first 3D point cloud with a 3D point spread function associated with a first type of 3D scanning sensor that represents the uncertainty of the spatial position of each measurement point. The first volume surface function incorporates the resolution of the first type of 3D scanning sensor. Performing the resolution-adaptive fusion also includes generating a second volume surface function by 3D convolving each of the plurality of measurement points of the second 3D point cloud with a 3D point spread function associated with a second type of 3D scanning sensor that represents the uncertainty of the spatial position of each measurement point. The second volume surface function incorporates the resolution of the second type of 3D scanning sensor.

[0006] According to the embodiment and any of the preceding embodiments, performing the resolution adaptive fusion further comprises: forming a 3D composite volume surface function by multiplying or adding a first volume surface function for the first type of 3D scanning sensor and a second volume surface function for the second type of 3D scanning sensor. By forming the 3D composite volume surface function, inaccurate point cloud data from one type of 3D scanning sensor is compensated by accurate point cloud data from the other type of scanning sensor.

[0007] According to an embodiment and any one of the preceding embodiments, wherein forming the 3D composite volume surface function comprises adding the first volume surface function and the second volume surface function in response to deactivating one of the 3D scanning sensors.

[0008] According to an embodiment and any one of the preceding embodiments, wherein forming the 3D composite volume surface function includes multiplying the first volume surface function and the second volume surface function so as to enhance the resolution for detecting obstacles in the environment associated with the vehicle compared to using only the volume surface function of one of the 3D scanning sensors.

[0009] According to an embodiment and any of the preceding embodiments, wherein performing the resolution adaptive fusion further comprises generating iso-contours of the 3D composite volume surface function by performing automatic edge-based thresholding to find optimal resolution adaptive iso-contours of the 3D composite volume surface function. The automatic edge-based thresholding is based on an optimization of a volume surface function edge map.

[0010] According to an embodiment and any one of the preceding embodiments, wherein performing automatic edge-based thresholding comprises incrementing a threshold within a preset range of values to determine a threshold that maximizes the number of edges in a two-dimensional (2D) edge map of contour lines of the 3D composite volume surface function.

[0011] According to an embodiment and any one of the preceding embodiments, performing the resolution adaptive fusion further comprises: resampling the contour lines of the 3D composite volume surface function on a uniform grid to form a fused, denoised and resolution optimized 3D point cloud.

[0012] According to an embodiment and any of the preceding embodiments, wherein the set of functions of the method and system further comprises rendering a representation of an environment associated with the vehicle using the fused, denoised and resolution optimized 3D point cloud.

[0013] According to an embodiment and any one of the preceding embodiments, wherein the set of functions of the method and system further comprises: detecting and avoiding obstacles for the vehicle using the fused, denoised and resolution optimized 3D point cloud.

[0014] According to an embodiment and any one of the preceding embodiments, wherein the set of functions of the method and system further includes: generating a series of fused, denoised and resolution-optimized 3D point clouds; and tracking a moving obstacle among obstacles using the series of fused, denoised and resolution-optimized 3D point clouds.

[0015] According to an embodiment and any one of the preceding embodiments, wherein the first type of 3D scanning sensor comprises one of a radar, a stereo vision sensor, a monocular vision sensor, or a LIDAR sensor, and wherein the second type of 3D scanning sensor comprises a sensor of a different type than the first type of 3D scanning sensor.

[0016] The features, functions, and advantages that have been discussed can be achieved independently in various embodiments or may be combined in yet other embodiments further details of which can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A is a schematic block diagram of an example of a vehicle including a system for obstacle detection and vehicle navigation using resolution-adaptive fusion of point clouds according to an embodiment of the present disclosure.

[0018] Figure 1B is a schematic block diagram of an example of a vehicle including a system for obstacle detection and vehicle navigation using resolution-adaptive fusion of point clouds according to another embodiment of the present disclosure.

[0019] Figure 2A and Figure 2B is a flowchart of an example of a method for obstacle detection and vehicle navigation using resolution-adaptive fusion of point clouds according to an embodiment of the present disclosure.

[0020] Figure 3A and Figure 3B yes Figure 2A and Figure 2B Illustration of the resolution adaptive fusion processing flow.

[0021] Figure 4A and Figure 4B Figure 1 shows examples of measurement points of different 3D point clouds and the associated resolution or measurement uncertainty for each measurement point.

[0022] Figure 5 is an illustration of a volume surface function that encompasses the sensor resolution formed by convolving the measurement points of a 3D point cloud with the point spread function (PSF) associated with the particular type of 3D scanning sensor that generated the 3D point cloud, in accordance with an embodiment of the present disclosure.

[0023] Figure 6 is a flow chart of an example of a method for performing automatic edge-based thresholding to find optimal resolution adaptive contours of a 3D composite volume surface function (VSF) according to an embodiment of the present disclosure.

[0024] Figure 7 yes Figure 6 Illustration of automatic edge-based thresholding to find the best-resolution adaptive contours of a 3D composite VSF.

[0025] Figure 8 is a diagram illustrating an example of a graph of changes in the total length of edges in a 2D projected edge map of contour lines of a VSF versus a threshold value according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following detailed description of the embodiments refers to the accompanying drawings, which illustrate specific embodiments of the present disclosure. Other embodiments with different structures and operations do not depart from the scope of the present disclosure. In different drawings, like reference numerals may refer to the same elements or components.

[0027] The present disclosure may be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute various aspects of the present disclosure.

[0028] A computer-readable storage medium can be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be understood as being a transient signal in itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0029] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0030] The computer-readable program instructions for performing the operation of the present disclosure can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages (including object-oriented programming languages such as Smalltalk, C++ and conventional process programming languages such as "C" programming language or similar programming languages). The computer-readable program instructions can be used as an independent software package completely on the user's computer, partially executed on the user's computer, partially on the user's computer and partially on a remote computer or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, the electronic circuit comprising, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit, to perform aspects of the present disclosure.

[0031] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0032] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in the flowcharts and / or block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct the computer, programmable data processing device, and / or other device to function in a specific manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in the flowcharts and / or block diagram blocks.

[0033] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus, so that a series of operational steps are performed on the computer, other programmable device, or other apparatus to produce a computer-implemented process, such that the instructions executed on the computer, other programmable device, or other apparatus implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0034] Figure 1A is a schematic block diagram of an example of a vehicle 100 including a system 102 for obstacle 104a or 104b detection and vehicle 100 navigation using resolution adaptive fusion 138 of 3D point clouds 106 and 108 according to an embodiment of the present disclosure. Figure 2A and Figure 2B An example of a method 200 for performing resolution adaptive fusion 138 is described. Figure 1A In the example shown in FIG, obstacles 104 include one or more fixed obstacles 104a and one or more mobile obstacles 104b in environment 110 associated with vehicle 100. According to various examples, vehicle 100 is an aircraft, an unmanned aerial vehicle (UAV), or other type of autonomous vehicle.

[0035] The system 102 includes a first type of 3D scanning sensor 112 and at least a second type of 3D scanning sensor 114. In other embodiments, the system 102 includes more than two types of 3D scanning sensors 112 and 114. Examples of types of 3D scanning sensors include, but are not limited to, any type of vision sensor, such as a stereo vision sensor or a monocular vision sensor, a light detection and ranging (LIDAR) sensor, a radar, and any other type of 3D sensing system that measures and tracks objects in a large volume of space. Each of the different types of 3D scanning sensors 112 and 114 collects one or more electronic images 118 and 120, respectively, of the environment 110 associated with the vehicle 100. As described in more detail herein, the electronic images 118 and 120 have different resolutions for different viewing directions and distances from the 3D scanning sensor 112 or 114 based on the sensor resolution of the particular type of 3D scanning sensor 112 and 114 for the different directions and distances. In Figure 1A In the example of FIG, environment 110 includes a ground plane 122, one or more fixed obstacles 104a, and one or more mobile obstacles 104b.

[0036] Each electronic image 118 includes a 3D point cloud 106, and each electronic image 120 includes a 3D point cloud 108. Each 3D point cloud 106 and 108 includes a plurality of measurement points 124, and each measurement point 124 includes point cloud data 126. Each measurement point 124 corresponds to a point 128 on a surface 130 of one of obstacles 104a and 104b in the environment 110 associated with the vehicle 100. The point cloud data 126 includes position or location information for the corresponding point 128 on the surface 130 of the obstacle 104a or 104b, as well as a reference to the position of the corresponding point 128. Figure 4A and Figure 4B and Figure 5 Describe the measurement uncertainty in more detail.

[0037] The system 102 also includes a processor 132 and a memory 134 associated with the processor 132. Figure 1A In the example of FIG, the memory 134 is shown as part of the processor 132. In other embodiments, the memory 134 is a separate component from the processor 132. The memory 134 includes computer readable program instructions 136, which, when executed by the processor 132, cause the processor 132 to perform a set of functions 137. The set of functions 137 defines the resolution adaptive fusion 138 process. As shown in FIG. Figure 2A and Figure 2BAs described in more detail, resolution adaptive fusion 138 combines 3D point clouds (such as point clouds 106 and 108) from multiple types of 3D scanning sensors (e.g., 3D scanning sensors 112 and 114) having different resolutions in different dimensions or directions into a single uniformly sampled point cloud, such as fused, denoised, and resolution-optimized 3D point cloud 140. The resolution of the fused, denoised, and resolution-optimized 3D point cloud 140 in each dimension is approximately equal to the resolution of the particular type of 3D scanning sensor 112 or 114 having the best resolution in that dimension or direction. Resolution adaptive fusion 138 uses sensor resolution models for different directions and distances to prioritize the most accurate data in creating the fused, denoised, and resolution-optimized 3D point cloud 140. As described in reference Figure 2A and Figure 2B As described in detail, the resolution adaptive fusion 138 incorporates a sensor resolution model to compensate for the poor resolution of a 3D scanning sensor 112 or 114 in one dimension by automatically suppressing the low resolution data and replacing it with data from another 3D scanning sensor 112 or 114 that performs better in that dimension. This enables high performance sensing in all dimensions by using different types of 3D scanning sensors 112 and 114. For example, at least a first type of 3D scanning sensor 112 has good performance in one or more dimensions, while at least a second type of 3D scanning sensor 114 has good performance in the dimensions where the first type of scanning sensor 114 has poor performance. As a further example, the resolution adaptive fusion 138 can replace LIDAR with vision and radar 3D scanning sensors because vision and radar have complementary performance in different dimensions and environmental conditions. Another advantage of the resolution adaptive fusion 138 is that all parameters can be calculated from the sensor resolution model or automatically determined by the system 102, which enables the resolution adaptive fusion 138 to automatically adapt to the environment 110. As shown in FIG. Figure 2A and Figure 2BIn more detail, resolution-adaptive fusion 138 3D-convolves each measurement point 124 of first 3D point cloud 106 with a 3D point spread function (PSF) 142 associated with first type 3D scanning sensor 112 to incorporate the sensor resolution of first type 3D scanning sensor 112 when generating a fused, noise-reduced, and resolution-optimized 3D point cloud 140. Resolution-adaptive fusion 138 also 3D-convolves each measurement point 124 of second 3D point cloud 108 with a 3D PSF 144 associated with second type scanning sensor 114 to incorporate the sensor resolution of second type 3D scanning sensor 114 when generating a fused, noise-reduced, and resolution-optimized 3D point cloud 140. The sensor resolution model is represented by 3D PSFs 142 and 144. PSFs 142 and 144 can vary spatially. For example, the resolution range of a stereo vision sensor depends on the range of each 3D measurement point 124 from the sensor.

[0038] According to various exemplary embodiments, vehicle 100 includes at least one of a perception system 146 and a display 148, a vehicle control system 150, and a mobile obstacle tracking system 152. Perception system 146 is configured to use the fused, de-noised, and resolution-optimized 3D point cloud 140 to detect and avoid obstacles 104a and 104b in environment 110 and to navigate vehicle 100. For example, where vehicle 100 is an aircraft, the fused, de-noised, and resolution-optimized 3D point cloud 140 is presented on display 148 to provide the flight crew with enhanced situational awareness of obstacles 104a and 104b, for example, in adverse conditions such as low visibility. In instances where vehicle 100 is an unmanned aerial vehicle or an autonomous vehicle, perception system 146 uses the fused, de-noised, and resolution-optimized 3D point cloud 140 to detect and avoid obstacles 104a and 104b.

[0039] The vehicle control system 150 is configured to use the fused, denoised, and resolution-optimized 3D point cloud 140 for navigation of the vehicle 100 and avoiding obstacles 104 a and 104 b .

[0040] The mobile obstacle tracking system 152 is configured to track one or more mobile obstacles 104b using a series of fused, noise-reduced, and resolution-optimized 3D point clouds 140. Figure 2A and Figure 2B An example of generating a series of fused, denoised, and resolution-optimized 3D point clouds for tracking the moving obstacle 104b is described in more detail.

[0041] Figure 1Bis a schematic block diagram of an example of a vehicle 101 according to another embodiment of the present disclosure, the vehicle 101 including a system 102 that uses resolution adaptive fusion 138 of point clouds 106 and 108 for obstacle 104a and 104b detection and navigation of the vehicle 101. The vehicle 101 is similar to Figure 1A 146, the vehicle 101 being located at least one of a perception system 146 and an associated display 148, a vehicle control system 150, and a mobile obstacle tracking system 152. In an embodiment, the vehicle 101 is an autonomous vehicle, such as an unmanned aerial vehicle (UAV). The vehicle 101 includes a transceiver 156 for transmitting the fused, noise-reduced, and resolution-optimized 3D point cloud 140 to another transceiver 158 at the ground station 154 for use by the perception system 146, the vehicle control system 150, and the mobile obstacle tracking system 152, similar to those described above. The transceiver 156 is further configured to receive control signals from the transceiver 158 at the ground station 154 to control the operation of the vehicle 101.

[0042] According to another embodiment, the processor 132 and associated memory 134 are also located at the ground station 154. In this embodiment, the transceiver 156 transmits the electronic images 118 and 120, including the respective 3D point clouds 106 and 108, to the transceiver 158 of the ground station 154. The ground station 154 then performs resolution adaptive fusion 138, and the ground station 154 transmits control signals to the vehicle 101 for navigation of the vehicle 101 and avoidance of obstacles 104a and 104b.

[0043] Figure 2A and Figure 2B is an embodiment of the present disclosure using 3D scanning sensors 112 and 114 ( Figure 1A and Figure 1B ) is used to perform resolution adaptive fusion 138 of the 3D point clouds 106 and 108 to detect obstacles 104a and 104b ( Figure 1A and Figure 1B ) detection and such Figure 1A and Figure 1B Flowchart of an example of a method 200 for vehicle navigation of a vehicle 100 or 101 in FIG. Figure 1A In the example of FIG. 2 , at least some of the functionality of the method 200 is implemented in the computer-readable program instructions 136 and the resolution-adaptive fusion 138 .

[0044] In block 202, a first type of 3D scanning sensor 112 ( Figure 1A and Figure 1B) performs a first scanning operation to collect one or more electronic images 118 of the environment 110 associated with the vehicle 100 or 101. In block 204, at least a second scanning operation is performed using the second type of 3D scanning sensor 114 to collect one or more electronic images 120. Each electronic image 118 from the first type of 3D scanning sensor 112 includes a first 3D point cloud 106, and each electronic image 120 from the second type of scanning sensor 114 includes a second 3D point cloud. As previously described, each 3D point cloud 106 and 108 includes a plurality of measurement points 124. Each measurement point 124 corresponds to a point 128 on the surface 130 of the obstacle 104a or 104b. Each measurement point 124 includes point cloud data 126. The point cloud data 126 includes at least information defining the location of the point 128 on the surface 130 of the obstacle 104a or 104b.

[0045] In block 206 , resolution adaptive fusion 138 is performed on at least the first 3D point cloud 106 and the second 3D point cloud 108 to generate a fused, denoised, and resolution optimized 3D point cloud 140 . Figure 2A and Figure 2B In the exemplary embodiment, resolution-adaptive fusion 138 in block 206 includes blocks 208-224. The fused, noise-reduced, and resolution-optimized 3D point cloud 140 represents the environment 110 associated with the vehicle 100 or 101. As previously described, the first 3D point cloud 106 is generated by a first-type 3D scanning sensor 112, and the second 3D point cloud 108 is generated by a second-type 3D scanning sensor 114. The second-type 3D scanning sensor 114 has a different resolution in each of a plurality of different measurement dimensions relative to the first-type 3D scanning sensor 112. As previously described, the fused, noise-reduced, and resolution-optimized 3D point cloud 140 is used to detect any obstacles 104a and 104b and to navigate the vehicle 100 or 101.

[0046] Typical combinations of at least a first type of 3D scanning sensor 112 and a second type of 3D scanning sensor 114 include, but are not limited to, vision sensors, radar, and LIDAR sensors. Each of these different types of 3D scanning sensors 112 and 114 has advantages, but also disadvantages that can be compensated for by one of the other types of 3D scanning sensors 112 and 114. Because no single type of 3D scanning sensor 112 or 114 has an ideal feature set, multiple different types of 3D scanning sensors 112 and 114 are utilized and fused together to form a unified representation of the environment 110 of the vehicle 100 or 101 for detecting and classifying obstacles 104a and 104b. For example, stereo or monocular vision 3D scanning sensors have good resolution in azimuth and elevation, and good range resolution at short distances, but degrade rapidly at longer ranges. Vision 3D scanning sensors are inexpensive and compact. LIDAR sensors have excellent range resolution and intermediate azimuth and elevation resolution, but they are bulky and expensive. Both vision and LIDAR sensors are affected by weather conditions. Radar has good range resolution, but poor azimuth and elevation resolution. Unlike the other two sensors, radar can also measure velocity towards the radar using Doppler shift and can operate in all weather conditions. Figure 1A and Figure 1B While the exemplary embodiments in the drawings show at least a first type of 3D scanning sensor 112 and a second type of 3D scanning sensor 114, other embodiments include more than two types of 3D scanning sensors with different resolution characteristics, such that point cloud data 126 from the different types of sensors can be used to generate a fused, denoised, and resolution-optimized 3D point cloud 140 using the resolution adaptive fusion 138 process described herein.

[0047] Also refer to Figure 3A and Figure 3B , Figure 3A and Figure 3B yes Figure 2A and Figure 2B 206 of the resolution adaptive fusion 138 process flow. Figure 2A In block 208 of the embodiment, a first volume surface function (VSF) 212 is generated by using a fast Fourier transform (FFT) in each of the three spatial dimensions to generate a 3D convolution 302 of each measurement point 124 of the plurality of measurement points 124 from the first 3D point cloud 106 with the associated 3D PSF 142 of the first type of 3D scanning sensor 112 representing the uncertainty in the spatial position of each measurement point 124 ( Figure 3A ). The first VSF 212 is combined with the resolution of the first type 3D scanning sensor 112.

[0048] In block 210, a second VSF 214 is generated by 3D convolving 304 each measurement point 124 from the plurality of measurement points 124 of the second 3D point cloud 108 with an associated 3D PSF 144 of the second type of 3D scanning sensor 114 representing the spatial position uncertainty of each measurement point 124 of the second 3D point cloud 108. The second VSF 214 incorporates the resolution of the second type of 3D scanning sensor 114.

[0049] Resolution adaptive fusion 138 assumes that sensor resolution models are available for the resolution of each 3D scanning sensor 112 and 114 at different viewing directions and distances. These resolution models can be determined analytically using a physical model of the sensor or empirically by measuring resolution targets at multiple locations, which are then interpolated to cover the entire measurement volume. Figure 4A and Figure 4B As shown, a resolution or measurement uncertainty value is associated with each measurement point 124 in 3D point clouds 106 and 108 . Figure 4A and Figure 4B 1 is a diagram of an example of measurement points 124 for different 3D point clouds 106 and 108 and the associated resolution or measurement uncertainty 402 for each measurement point 124. Each measurement point 124 is defined by its position and its associated measurement uncertainty 3D point spread function (PSF) 142 or 144, which is formed by the Gaussian product of each dimension. The PSF axes correspond to the range of the measurement point 128 from the obstacle 104a or 104b and the resolution or standard deviation σ in the two lateral directions. x ,σ y and σ z ( Figure 5 ), as seen from 3D scanning sensor 112 or 114 at measurement point 128. The orientation of PSFs 142 and 144 is represented by two angles: azimuth (θ) and elevation (φ). A total of eight parameter values are associated with each measurement point 124 to represent the position and measurement uncertainty of point 124 at any direction and distance. PSFs 142 or 144 may vary with position within the measurement volume, depending on the characteristics of the sensor.

[0050] Since the 3D scanning sensors 112 and 114 have different resolutions at different directions and distances, the resolution adaptive fusion 138 is configured to preferentially utilize the point cloud data 126 with the best resolution from each 3D scanning sensor 112 and 114. The basic principle of the resolution adaptive fusion 138 is described in Figure 5. The resolution adaptive fusion 138 uses an intermediate "implicit" volume representation 502 of evidence of the object surface in the spatial volume to generate a new, higher accuracy point cloud 504 with reduced noise, which is sampled on a uniform spatial grid. Unlike the "explicit" representation of a surface as a list of points on the surface with 3D coordinates, the implicit approach represents the surface in terms of a function defined in 3D space. For example, one possible implicit representation is a 3D function whose value for each 3D voxel is given by the density of points at that location. Another implicit representation utilized by the resolution adaptive fusion 138 is to define the surface as a contour or level set of the 3D function, which models the resolution of the 3D scanning sensors 112 and 114 at different locations and in different directions. In resolution adaptive fusion 138, a 3D function is formed by convolving the measurement point 124 measured by each 3D scanning sensor 112 and 114 with the associated 3D PSF 142 and 144 of each 3D scanning sensor 112 and 114 to form a volume surface function (VSF) 212 and 214. The PSF 142 and 144 represent the uncertainty of the position of each measurement point 124 in all directions and can be modeled as a Gaussian product, where the position, shape, and orientation of the PSF 142 and 144 reflect the resolution characteristics of the specific 3D scanning sensor 112 and 114. The PSF 142 and 144 are derived from a priori sensor models or are empirically measured based on data. The 3D convolutions 302 and 304 replace each measurement point 124 with the 3D PSF 142 or 144 of the associated sensor at that location, as shown in FIG. Figure 5 is shown and is efficiently implemented using the Fast Fourier Transform.

[0051] The VSF 212 and 214 are 3D voxel-based 3D volumetric representations of the 3D point cloud 106 or 108, respectively, and are a measure of the presence of the surface 130 of the obstacle 104a or 104b at each 3D spatial location, combined with the resolution of the corresponding 3D scanning sensor 112 or 114. The VSF 212 and 214 are similar to how a computed axial tomography (CAT) scan is a 3D function that maps the density of human tissue. As more points 124 are measured, the most likely surface location will correspond to the voxel with the highest VSF value due to the overlap of the PSFs from multiple measured points 124.

[0052] In block 216, the first type of 3D scanning sensor 112 ( Figure 1A and Figure 1B ) and for the first VSF 212 of the second type 3D scanning sensor 114 ( Figure 1A and Figure 1B) to form a 3D composite VSF 218. By forming the 3D composite VSF 218, inaccurate point cloud data 126 from one type of 3D scanning sensor 112 or 114 is compensated for by accurate point cloud data 126 from the other type of 3D scanning sensor 112 or 114. The 3D composite VSF 218 is formed by adding the first VSF 212 and the second VSF 214 in response to a condition that renders one of the 3D scanning sensors 112 or 114 invalid. The 3D composite VSF 218 is formed by multiplying the first VSF 212 and the second VSF 214 so as to enhance the resolution used to detect obstacles 104a or 104b in the environment 110 associated with the vehicle 100 or 101, compared to the VSF 212 or 214 using only the 3D scanning sensor 112 or 114. Since the composite resolution in each dimension will be the resolution of the 3D scanning sensor 112 or 114 with the higher resolution in that dimension, the multiplication of the VSFs 212 and 214 is appropriate when all obstacles in the environment are detected by two or more 3D scanning sensors 112 and 114. However, if only one 3D scanning sensor 112 or 114 detects obstacle 104a or 104b, the VSF 212 or 214 of the other sensor 112 or 114 will not cover that obstacle 104a or 104b, and the product of the VSFs 212 and 214 for that obstacle 104a or 104b will be zero. However, if the VSFs 212 and 214 are added together, obstacles 104a or 104b detected by only one 3D scanning sensor 112 or 114 will be represented by the 3D composite VSF 218. The resolution of obstacles 104 a and 104 b detected by multiple 3D scanning sensors 112 and 114 may not improve as much as in the multiplication case, but simulations show that the performance improvement is still significant. A preferred mode of operation may be to use the additive VSF to initially detect obstacles 104 a and 104 b, and then use the multiplicative VSF to refine the resolution of obstacles 104 a and 104 b to calculate the multiplication and addition results of VSFs 212 and 214.

[0053] exist Figure 2B In block 220 of , by performing automatic edge-based thresholding 306 ( Figure 3B ) to find the best resolution adaptive contour line 222 of the 3D composite VSF 218 to generate the contour line 222 of the 3D composite VSF 218 (see also Figure 3B ). Automatic edge-based thresholding 306 is based on VSF edge map optimization. Figure 6 and Figure 7 An example of a method for automatic edge-based thresholding is described. Figure 7 and Figure 8 In more detail, performing automatic edge-based thresholding 306 includes incrementing thresholds 702a-702c within a preset range of values to determine a threshold that maximizes the number of edges in a two-dimensional (2D) edge map 704b of the contour lines 222 of the 3D composite VSF 218 (in Figure 7 702b in the example).

[0054] In block 224, performing resolution adaptive fusion 138 also includes performing resolution adaptive fusion 138 on the uniform grid 308 ( Figure 3B ) resamples the contour lines 222 of the 3D composite VSF 218 to form a fused, denoised and resolution-optimized 3D point cloud 140. According to an example, the resampling includes: voxel-based point cloud resampling 310 of the contour lines 222 of the 3D composite VSF 218 ( Figure 3B ) to provide a fused, denoised, and resolution-optimized 3D point cloud 140. Voxel-based point cloud resampling uses as input the point cloud points (measurement points) 124 measured by sensors 112 and 114, which are irregularly distributed in 3D space due to the local topography of the sensing environment. A 3D composite VSF 218 is a continuous function defined in 3D space and is generated by convolving each measurement point 124 in each point cloud 106 and 108 with the local PSF 142 and 144 of the associated sensor. The 3D volume domain of the 3D composite VSF 218 is subdivided into regularly spaced continuous voxels, whose values are the local average of the 3D composite VSF 218 over the voxel volume, thereby performing spatial sampling of the 3D composite VSF 218. Thresholding the 3D composite VSF 218 produces a contour 222 surface of the 3D composite VSF 218 sampled at the voxel location. Voxels above the threshold are then replaced with points centered around those voxels above the threshold. These new points form a resampled point cloud on a regularly spaced 3D grid, which is a fused, denoised, and resolution-optimized point cloud 140 formed from the point clouds 106 and 108 measured by the individual sensors.

[0055] Depending on the embodiment, the method 200 returns to blocks 202 and 204, and the method 200 repeats as described above. The method 200 is repeated a predetermined number of times or continuously during the operation of the vehicle 100 or 101 to generate a series of fused, noise-reduced, and resolution-optimized 3D point clouds 140. In block 226, the series of fused, noise-reduced, and resolution-optimized 3D point clouds 140 are used to navigate the vehicle 100 or 101 and detect and avoid stationary obstacles 104a and moving obstacles 104b in block 226.

[0056] In block 228, the mobile obstacle 104b is tracked using the series of fused, noise-reduced, and resolution-optimized 3D point clouds 140. In block 230, a representation of the environment 110 associated with the vehicle 100 or 101 is presented on a display 148 using the fused, noise-reduced, and resolution-optimized 3D point cloud 140 or the series of fused, noise-reduced, and resolution-optimized 3D point clouds 140.

[0057] Figure 6 is a method for performing automatic edge-based thresholding 306 ( Figure 3B ) to find the best resolution adaptive contour 222 of the 3D composite VSF 218. The automatic edge-based thresholding 306 is based on the method 600 performed and in Figure 7 The volume surface function (VSF) edge map optimization 700 is shown in FIG. Figure 6 In block 602 , the threshold is set to zero (0). In block 604 , contour lines 222 of the 3D composite VSF 218 are generated by setting all voxels with a value above the threshold to 1 and to 0 otherwise.

[0058] In block 606, the contour lines 222 of the 3D composite VSF 218 are projected in the Z direction to form a two-dimensional (2D) image 706 ( Figure 7 ).

[0059] In block 608, 2D edge detection is performed and binary maps 704a-704c of the edges are formed. Binary maps 704a-704c can be thought of as images where pixel values are 1 at edge locations and 0 at other locations. Therefore, summing all pixels in the binary map image is a measure of the number of edge pixels.

[0060] In block 610, all pixels in the binary map (also referred to as the binary edge map) 704a-704c are summed to measure the number of edges. The sum of all edges is saved.

[0061] In block 612, a determination is made as to whether the threshold is equal to one (1). If the threshold is not equal to one (1), the value of the threshold is increased by a preset amount that is less than one (1), and the method 600 returns to block 604. The method 600 then proceeds as previously described.

[0062] If the threshold in block 612 is equal to one (1), the method 600 proceeds to block 616. In block 616, the threshold is set to a value that maximizes the number of edges. In block 618, the set threshold is used to generate the optimal contour 222 of the 3D composite VSF 218.

[0063] Also refer to Figure 7 . Figure 7 yes Figure 6 3D composite VSF 218 to find the best resolution adaptive contour 222. Figure 7 As shown, as the threshold values 702a-702c increase from a smaller value, i.e., 0.1, to a larger threshold value, the 2D projection of the contour line 222 on the XY plane ( Figure 6 6) consists of a single large 3D blob 708a, which then separates into smaller 3D blobs 708b-708c centered at each obstacle 104a or 104b as the thresholds 702a-702c are increased (block 614). These blobs 708a-708c then continue to shrink until a certain threshold, such as 702b, is reached, at which point the blob 708b breaks down into 3D "islands" centered at each measurement point 124. This threshold 702b is optimal in the sense that it is the maximum threshold that minimizes the volume of the contour line 222 (thus improving resolution) while still maintaining complete coverage of the obstacle 104a or 104b. As the thresholds 702a-702c are increased beyond this point, the blobs 708a-708c will shrink until they virtually disappear (blob 708c). A simple computational variable that can be used to detect the optimal threshold is the value of the contour line 222 at Figure 7 The total length of the edges in the projected edge map 706 on the 2D plane or XY plane shown. By increasing the threshold values 702a-702c within the threshold range and selecting the threshold value that maximizes the total edge length of the 3D composite VSF contour line 222, as shown in FIG. Figure 7 and Figure 8 As shown, thresholds 702a-702c may be set automatically.

[0064] Figure 8 is a 2D projected edge map 706 of the contour lines 222 of the VSF according to an embodiment of the present disclosure ( Figure 7 ) is an illustration of an example of a graph 800 of the change in the total length of the edges in the image versus the threshold value. Figure 8 As shown, the optimal threshold 702b for combining the sensor data is the threshold that maximizes the total edge length.

[0065] Although Figure 2A and Figure 2BThe exemplary method 200 in FIG. 1 includes two scanning operations to generate the first and second 3D point clouds 106 and 108. According to other embodiments, more than two scanning operations are performed. In another embodiment, at least a third scanning operation is performed by a third type of 3D scanning sensor to generate at least a third 3D point cloud. Resolution-adaptive fusion 138 is then performed using the at least three 3D point clouds. Resolution-adaptive fusion 138 can be performed using any number of 3D point clouds generated by different types of 3D scanning sensors to generate a fused, noise-reduced, and resolution-optimized 3D point cloud 140.

[0066] From the embodiments described herein, one skilled in the art will recognize that resolution adaptive fusion 138 is applicable to any platform that utilizes multiple sensors to sense a 3D environment for applications such as obstacle detection and navigation in taxiing aircraft, drones, and other autonomous vehicles. Resolution adaptive fusion 138 improves the 3D resolution of a sensor system by enabling one sensor to compensate for the poor resolution of a second sensor in a particular measurement direction or the inability of the second sensor to function effectively under current weather and / or lighting conditions. For example, resolution adaptive fusion 138 can be configured to automatically switch between low-cost and compact vision and radar sensors for different ranges and azimuth / elevation angles to create a high-resolution fused 3D point cloud using the best sensor for each dimension. One potential application is to use a combination of radar and vision sensors to reduce or eliminate the need for expensive, low-resolution, and LIDAR sensors.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, fragment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the box may not occur in the order indicated in the figure. For example, depending on the functions involved, the two boxes shown in succession may actually be executed substantially simultaneously, or sometimes these boxes may be executed in the opposite order. It should also be noted that each box of the block diagram and / or flowchart and the combination of the boxes of the block diagram and / or flowchart may be implemented by a hardware-based dedicated system for the specific purpose of performing a specified function or action or performing a combination of hardware and computer instructions for a special purpose.

[0068] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present disclosure. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "comprises", "includes", "contains", and / or "covers" specify the presence of the features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0069] All means or steps plus corresponding structures, materials, acts, and equivalents of function elements in the appended claims are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments.

[0070] Although specific embodiments have been shown and described herein, it will be understood by those skilled in the art that any arrangement designed to achieve the same purpose may be substituted for the specific embodiments shown, and that these embodiments have other applications in other environments. This application is intended to cover any adaptations or variations. The appended claims are in no way intended to limit the scope of the embodiments of the present disclosure to the specific embodiments described herein.

Claims

1. A method for obstacle detection and vehicle navigation using resolution adaptive fusion, the method comprising: performing, by a processor, resolution-adaptive fusion of at least a first 3D point cloud and a second 3D point cloud to generate a fused, noise-reduced, and resolution-optimized 3D point cloud representing an environment associated with the vehicle, wherein the first 3D point cloud was generated by a first type of 3D scanning sensor and the second 3D point cloud was generated by a second type of 3D scanning sensor, wherein the second type of 3D scanning sensor has a different resolution in each of a plurality of different measurement dimensions relative to the first type of 3D scanning sensor; Wherein, performing resolution adaptive fusion further includes: generating a first volume surface function, the first volume surface function incorporating a resolution of the first type of 3D scanning sensor; generating a second volume surface function incorporating a resolution of the second type of 3D scanning sensor; and forming a 3D composite volume surface function by multiplying or adding the first volume surface function for the first type of 3D scanning sensor and the second volume surface function for the second type of 3D scanning sensor, Among them, forming a 3D composite volume surface function includes: forming a 3D composite multiplied volume surface function by multiplying the first volume surface function for the first type of 3D scanning sensor and the second volume surface function for the second type of 3D scanning sensor; and forming a 3D composite additive volume surface function by adding the first volume surface function and the second volume surface function, wherein the 3D composite additive volume surface function is used to detect the obstacle and the 3D composite multiplicative volume surface function is used to refine the resolution of the obstacle; and Obstacles are detected and the vehicle is navigated using the fused, denoised, and resolution-optimized 3D point cloud.

2. The method according to claim 1, wherein Performing the resolution adaptive fusion includes: generating a first volume surface function by 3D convolving each measurement point of a plurality of measurement points from the first 3D point cloud with an associated 3D point spread function of the first type of 3D scanning sensor representing a spatial position uncertainty of each measurement point, and A second volume surface function is generated by 3D convolving each measurement point from a plurality of measurement points of the second 3D point cloud with an associated 3D point spread function of the second type of 3D scanning sensor representing the spatial position uncertainty of each measurement point.

3. The method according to claim 1, wherein By forming the 3D composite volume surface function, inaccurate point cloud data from one type of 3D scanning sensor will be compensated by accurate point cloud data from another type of scanning sensor.

4. The method according to claim 1, wherein In response to deactivating one of the first type of 3D scanning sensor and the second type of 3D scanning sensor, the first volume surface function and the second volume surface function are added.

5. The method according to claim 1, wherein Performing the resolution adaptive fusion also includes generating contour lines of the 3D composite volume surface function by performing automatic edge-based thresholding to find the optimal resolution adaptive contour lines of the 3D composite volume surface function, wherein the automatic edge-based thresholding is based on volume surface function edge map optimization.

6. The method according to claim 5, wherein: Performing the automatic edge-based thresholding includes incrementing a threshold within a preset range of values to determine a threshold that maximizes a number of edges in a two-dimensional (2D) edge map of contours of the 3D composite volume surface function.

7. The method according to claim 5, wherein: Performing the resolution adaptive fusion further includes resampling the contours of the 3D composite volume surface function on a uniform grid to form the fused, denoised and resolution optimized 3D point cloud.

8. The method of claim 1, further comprising using the fused, denoised, and resolution-optimized 3D point cloud to render a representation of an environment associated with the vehicle.

9. The method according to claim 8, further comprising: Obstacles for the vehicle are detected and avoided using the fused, denoised, and resolution-optimized 3D point cloud.

10. The method according to claim 1, further comprising: Generate a series of fused, denoised and resolution-optimized 3D point clouds; and A moving obstacle among the obstacles is tracked using the series of fused, denoised and resolution-optimized 3D point clouds.

11. The method according to claim 1, wherein The first type of 3D scanning sensor comprises a radar, a stereo vision sensor, a monocular vision sensor, or a LIDAR sensor, and wherein the second type of 3D scanning sensor comprises a different sensor than the first type of 3D scanning sensor.

12. A system for obstacle detection and vehicle navigation using resolution adaptive fusion, the system comprising: processor; and a memory associated with the processor, wherein the memory includes computer-readable program instructions that, when executed by the processor, cause the processor to perform a set of functions including: performing resolution-adaptive fusion of at least a first 3D point cloud and a second 3D point cloud to generate a fused, noise-reduced, and resolution-optimized 3D point cloud representing an environment associated with the vehicle, wherein the first 3D point cloud was generated by a first type of 3D scanning sensor and the second 3D point cloud was generated by a second type of 3D scanning sensor, wherein the second type of 3D scanning sensor has a different resolution in each of a plurality of different measurement dimensions relative to the first type of 3D scanning sensor, wherein performing resolution-adaptive fusion further comprises: generating a first volume surface function, the first volume surface function incorporating a resolution of the first type of 3D scanning sensor; generating a second volume surface function incorporating a resolution of the second type of 3D scanning sensor; and forming a 3D composite volume surface function by multiplying or adding the first volume surface function for the first type of 3D scanning sensor and the second volume surface function for the second type of 3D scanning sensor, Among them, forming a 3D composite volume surface function includes: forming a 3D composite multiplied volume surface function by multiplying the first volume surface function for the first type of 3D scanning sensor and the second volume surface function for the second type of 3D scanning sensor; and forming a 3D composite additive volume surface function by adding the first volume surface function and the second volume surface function, wherein the 3D composite additive volume surface function is used to detect the obstacle and the 3D composite multiplicative volume surface function is used to refine the resolution of the obstacle; and Obstacles are detected and the vehicle is navigated using the fused, denoised, and resolution-optimized 3D point cloud.

13. The system according to claim 12, wherein: Performing the resolution adaptive fusion includes: generating a first volume surface function by 3D convolving each measurement point from a plurality of measurement points of the first 3D point cloud with a 3D point spread function associated with the first type of 3D scanning sensor representing an uncertainty in the spatial position of each measurement point; and A second volume surface function is generated by 3D convolving each measurement point from a plurality of measurement points of the second 3D point cloud with an associated 3D point spread function of the second type of 3D scanning sensor representing the spatial position uncertainty of each measurement point.

14. The system according to claim 12, wherein: By forming the 3D composite volume surface function, inaccurate point cloud data from one type of 3D scanning sensor will be compensated by accurate point cloud data from another type of scanning sensor.

15. The system according to claim 12, wherein: Forming the 3D composite volume surface function includes adding the first volume surface function and the second volume surface function in response to deactivating one of the first type of 3D scanning sensor and the second type of 3D scanning sensor.

16. The system of claim 12, wherein: Performing the resolution adaptive fusion also includes generating contour lines of the 3D composite volume surface function by performing automatic edge-based thresholding to find the optimal resolution adaptive contour lines of the 3D composite volume surface function, wherein the automatic edge-based thresholding is based on volume surface function edge map optimization.

17. The system according to claim 16, wherein: Performing the automatic edge-based thresholding includes incrementing a threshold within a preset range of values to determine a threshold that maximizes a number of edges in a two-dimensional (2D) edge map of contours of the 3D composite volume surface function.

18. The system according to claim 16, wherein: Performing the resolution adaptive fusion further includes resampling the contours of the 3D composite volume surface function on a uniform grid to form the fused, denoised and resolution optimized 3D point cloud.

Images