Improved navigation and localization using surface penetrating radar and deep learning

Abstract

Deep learning is used to improve or measure the performance of a surface penetrating radar (SPR) system for localization or navigation. A vehicle can employ a terrain monitoring system that includes a SPR for obtaining SPR signals as the vehicle travels along a route. An on-board computer includes a processor and electronically stored instructions that are executable by the processor, which can analyze the acquired SPR images and compute a georecognition of subsurface structures therein by using the acquired images as inputs to a predictor that has been computationally trained to recognize subsurface structures in SPR images.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 900,098, filed September 13, 2019, and incorporates by reference herein in its entirety. Technical Field

[0003] The present invention relates generally to vehicle positioning and navigation, and more generally to improvements in accuracy and system monitoring using deep learning techniques. Background Art

[0004] Surface-penetrating radar (SPR) systems have been used for navigation and vehicle positioning; see, for example, U.S. Patent No. 8,949,024, the entire disclosure of which is incorporated herein by reference. SPR can be used in environments such as cities where multipath or shadows would degrade GPS accuracy, or as an alternative to optical sensing methods that cannot tolerate darkness or changing scene lighting, or where their performance may be adversely affected by changing weather conditions.

[0005] In particular, SPR can be used to acquire scans of both surface and subsurface features as a vehicle traverses terrain. These scans can then be compared with reference scans previously acquired in the same environment to localize the vehicle within that environment. If the reference scans have been geo-tagged, the vehicle's absolute position can be determined from them.

[0006] Scan data comparison can be a registration process based on, for example, correlation; see, for example, U.S. Patent No. 8,786,485, the entire disclosure of which is incorporated herein by reference. Although SPR positioning based on reference scan data overcomes the limitations of the above-mentioned conventional techniques, SPR sensors are not foolproof, and the registration process inevitably has a certain degree of error. For example, unclear ground conditions, SPR sensor aging or failure, vehicle speed, or changes in environmental conditions such as wind speed or temperature may cause errors.

[0007] Therefore, measures are needed to improve the accuracy of SPR-based positioning systems, minimize the occurrence of error conditions, and assess the reliability of real-time positioning estimates. Summary of the Invention

[0008] Embodiments of the present invention use deep learning to improve or measure the performance of SPR systems for positioning or navigation. The term "deep learning" refers to a machine learning algorithm that uses multiple layers to gradually extract higher-level features from raw images. Deep learning typically involves neural networks, which process information in a manner similar to the human brain. Such networks include a large number of highly interconnected processing elements (neurons) that work in parallel to solve a specific problem. Neural networks learn by example; they must be properly trained with carefully collected and planned training examples to ensure high performance levels, reduce training time, and minimize system bias.

[0009] Convolutional neural networks (CNNs) are commonly used to classify images or identify (and classify) objects depicted in image scenes. For example, a self-driving vehicle application might use a CNN in its computer vision module to identify traffic signs, cyclists, or pedestrians in the vehicle's path. CNNs extract features from input images using convolutions, which preserve the spatial relationships between pixels but facilitate learning image features using small blocks of input data. Neural networks learn from examples, so images can be labeled as containing or not containing features of interest. (Autoencoders can learn without labels.) For the system to perform reliably and efficiently, examples must be carefully selected and typically in large quantities.

[0010] Thus, in a first aspect, the present invention relates to a method for detecting and identifying subsurface structures. In various embodiments, the method comprises the steps of acquiring an SPR image, and computationally identifying subsurface structures in the acquired image by using the acquired image as input to a predictor, the predictor having been computationally trained to identify subsurface structures in the SPR image.

[0011] In some embodiments, the method further comprises acquiring additional SPR images during the traversal of the route; identifying, by the predictor, subsurface features in the SPR images that the predictor has been trained to identify; and associating the features identified in the images with ground coordinates corresponding to the time when the images were acquired, and generating, based thereon, an electronic map of the subsurface structure corresponding to the identified features.

[0012] In various embodiments, the method further includes the steps of acquiring additional SPR images during the vehicle's traverse of the route; identifying, by the predictor, subsurface features in the SPR images that the predictor has been trained to identify; associating the identified subsurface features with corresponding ground coordinates; and navigating the vehicle based at least in part on the identified subsurface features and their ground coordinates.

[0013] In another aspect, the present invention relates to a system for detecting and identifying subsurface structures. In various embodiments, the system includes an SPR system for acquiring SPR images, and a computer including a processor and electronically stored instructions executable by the processor for analyzing the acquired SPR images and computationally identifying subsurface structures therein by using the acquired images as input to a predictor that has been computationally trained to identify subsurface structures in the SPR images.

[0014] In any of the foregoing aspects, the predictor can be a neural network, such as a convolutional neural network or a recurrent neural network.

[0015] Yet another aspect of the present invention relates to a vehicle that, in various embodiments, includes an SPR system for acquiring SPR images while the vehicle is traveling, and a computer including a processor and electronically stored instructions, the instructions executable by the processor for analyzing the acquired SPR images and computationally identifying subsurface structures therein by using the acquired images as input to a predictor that has been computationally trained to identify subsurface structures in the SPR images.

[0016] In various embodiments, the computer is configured to associate the identified features in the image with ground coordinates corresponding to the time when the image was obtained, and based on this, generate an electronic map of the underground structure corresponding to the identified features. Alternatively or additionally, the computer can be configured to associate the identified features in the image with the ground coordinates corresponding thereto, and navigate the vehicle based at least in part on the identified underground features and their ground coordinates.

[0017] As used herein, the term "substantially" refers to ±10% of the volume of tissue, in some embodiments, ±5% of the volume of tissue. "Clinically significant" means that a clinician believes that an undesirable (sometimes referring to a lack of need) effect on tissue is significant, for example, causing damage. Throughout the specification, reference to "an example," "an example," "an embodiment," or "an embodiment" refers to references to specific features, structures, or characteristics described in conjunction with the example that are included in at least one example of the technical solution of the present invention. Therefore, the phrases "in one example," "in an example," "an embodiment," or "an embodiment" that appear throughout the specification do not necessarily all refer to the same example. In addition, specific features, structures, routines, steps, or characteristics can be combined in any suitable manner in one or more examples of the technical solution of the present invention. The titles provided herein are for convenience only and are not intended to limit or interpret the scope or meaning of the claimed technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The foregoing and the following detailed description will be more readily understood when taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1A An exemplary traveling vehicle including a terrain monitoring system according to an embodiment of the present invention is schematically illustrated.

[0020] Figure 1B Schematically illustrates an alternative configuration where the antenna of the terrain monitoring system is closer to or in contact with the road surface.

[0021] Figure 2 An exemplary terrain monitoring system according to an embodiment of the present invention is schematically illustrated.

[0022] Figure 3 Schematically illustrating an exemplary architecture in which a CNN or other form of deep learning is integrated with an SPR system. DETAILED DESCRIPTION

[0023] First reference Figure 1A , which shows an exemplary vehicle 102 traveling on a predetermined route 104; the vehicle 102 is equipped with a terrain monitoring system 106 for vehicle navigation according to the present application. In various embodiments, the terrain monitoring system 106 includes an SPR navigation and control system 108 having a ground penetrating radar (GPR) antenna array 110 fixed to the front (or any suitable portion) of the vehicle 102. The GPR antenna array 110 is generally oriented parallel to the ground surface and extends perpendicular to the direction of travel. In alternative configurations, the GPR antenna array 110 is closer to or in contact with the road surface ( Figure 1B ). In one embodiment, the GPR antenna array 110 includes a linear configuration of spatially invariant antenna elements for transmitting GPR signals to the road; the GPR signals can propagate through the road surface to the underground region and reflect in an upward direction. The reflected GPR signals can be detected by receiving antenna elements in the GPR antenna array 110. In various embodiments, the detected GPR signals are then processed and analyzed to generate one or more SPR images (e.g., GPR images) of the underground region along the trajectory of the vehicle 102. If the SPR antenna array 110 is not in contact with the ground, the strongest echo signal received may be a reflection caused by the road surface. Therefore, the SPR image can include surface data, i.e., data of the interface of the underground region with the air or local environment. Suitable GPR antenna configurations and systems for processing GPR signals are described, for example, in U.S. Patent No. 8,949,024, the entire disclosure of which is incorporated herein by reference.

[0024] For navigation, the SPR image is compared with an SPR reference image previously acquired and stored for an underground region that at least partially overlaps the underground region of the defined route. The image comparison can be a registration process based on correlations such as those described in the aforementioned '485 patent. The position of the vehicle 102 and / or the terrain conditions of the route 104 can then be determined based on the comparison. In some embodiments, the detected GPR signal is combined with other real-time information (such as weather conditions, electro-optical (EO) images, vehicle health monitoring using one or more sensors employed in the vehicle 102, and any suitable inputs) to estimate the terrain conditions of the route 104.

[0025] Figure 2 An exemplary navigation and control system (e.g., SPR system 108) implemented in vehicle 102 is shown for navigating travel based on SPR images. SPR system 108 may include a user interface 202, through which a user may input data to define a route or select a predetermined route. SPR images may be retrieved from an SPR reference image source 204 according to the route. For example, SPR reference image source 204 may be a local mass storage device, such as a flash drive or hard drive, etc. Alternatively or additionally, SPR reference image source 204 may be cloud-based (i.e., supported and maintained on a network server) and may be remotely accessed based on the current position determined by GPS. For example, local data storage may include an SPR reference image corresponding to the vicinity of the current position of the vehicle, and periodic updates may be retrieved to refresh data as the vehicle travels.

[0026] The SPR system 108 also includes a mobile SPR system (“mobile system”) 206 having an SPR antenna array 110. Transmit operations of the mobile SPR system 206 are controlled by a controller (e.g., a processor) 208, which also receives return SPR signals detected by the SPR antenna array 110. The controller 208 generates an SPR image of the road surface and / or the subsurface region below the road surface beneath the SPR antenna array 110.

[0027] The SPR image includes features representing structures and objects within the underground area and / or on the road surface, such as rocks, roots, boulders, pipes, voids, and soil layers, as well as other features representing changes in soil or material properties in the underground / surface area. In various embodiments, the registration module 210 compares the SPR image provided by the controller 208 with the SPR image retrieved from the SPR reference image source 204 to locate the vehicle 102 (e.g., by determining the offset of the vehicle relative to the nearest point on the route). In various embodiments, the position information determined during the registration process (e.g., offset data or position error data) is provided to the conversion module 212, which creates a position map for navigating the vehicle 102. For example, the conversion module 212 can generate GPS data that is corrected for the vehicle's position deviation from the route.

[0028] Alternatively, the conversion module 212 can retrieve an existing map from a map source 214 (e.g., other navigation systems such as GPS, or a mapping service) and then locate the obtained location information to the existing map. In one embodiment, the location map of the predefined route is stored in a database 216 in a system memory and / or a storage device accessible to the controller 208. Additionally or alternatively, the location data of the vehicle 104 can be combined with data provided by an existing map (e.g., a map provided by Google Maps) and / or one or more other sensors or navigation systems such as an inertial navigation system (INS), a GPS system, a sound navigation and ranging (SONAR) system, a lidar system, a camera, an inertial measurement unit (IMU) and an auxiliary radar system, one or more vehicle dead reckoning sensors (based on, for example, steering angle and tire odometers) and / or a suspension sensor to guide the vehicle 102. For example, the controller 112 can locate the obtained SPR information to an existing map generated using GPS. Methods for using the SPR system for vehicle navigation and positioning are described, for example, in the aforementioned '024 patent.

[0029] An exemplary architecture for integrating deep learning with the SPR navigation and control module 108 is shown in Figure 3As shown in . As described above, the system may include various sensors 310 deployed within the relevant vehicle. These include SPR sensors, but may also include sensors for identifying conditions related to the operation of one or more deep learning modules 315, as described below. Sensors 310 can also detect external conditions that may affect system performance and accuracy and trigger mitigation strategies. For example, as described in U.S. Serial No. 16 / 929,437 filed on July 15, 2020 and incorporated herein by reference in its entirety, sensors 310 can detect dangerous terrain conditions (combined with SPR measurements of subsurface features, which may also trigger hazard warnings). In response, system 108 can update map database 216 accordingly and, in some embodiments, issue a warning to local authorities. Sensors can also capture conditions related to the reliability of position estimates. For example, additional sensors 317 can sense environmental conditions (e.g., wind speed and / or temperature), and hardware monitoring sensors can sense vehicle performance parameters (e.g., speed) and / or vehicle health parameters (e.g., tire pressure and suspension performance) and / or SPR sensor parameters (e.g., sensor health or other performance indicators). The sensor data may be filtered and conditioned by appropriate hardware and / or software modules 323 , as is conventional in the art.

[0030] A number of software subsystems implemented as instructions stored in computer memory 326 are executed by a conventional central processing unit (CPU) 330. The CPU 330 may be used for the deep learning functionality described below or may also operate the controller 208 (see Figure 2 An operating system (e.g., such as MICROSOFT WINDOWS, UNIX, LINUX, iOS, or ANDROID) provides low-level system functions such as file management, resource allocation, and message routing from hardware devices to software subsystems and from software subsystems to hardware devices.

[0031] A filtering and adjustment module 335 suitable for a deep learning submodule (such as a CNN) is also implemented as a software subsystem. For example, if one of the submodules 315 is a CNN, the SPR image can be pre-processed by adjusting the input size to the CNN, denoising, edge smoothing, sharpening, etc. Depending on the selected deep learning submodule 315, the generated output can be post-processed by a post-processing module 338 for positioning, as well as generating metrics such as health status estimation, warnings, and map update information. Post-processing refers to the operation of formatting and adjusting the output from the deep learning submodule 315 into a format and value that can be used for positioning estimation. Post-processing can include any statistical analysis required to merge the outputs of the various deep learning modules into a single stream of values ​​that can be used for downstream processing (e.g., averaging the outputs of multiple deep learning modules), changing the data type so that the output can be used in the file format, network protocol, and API required for downstream processing. In addition, post-processing can include converting the deep learning module output into a traditional, non-deep learning algorithm required for position estimation (e.g., converting the probability density function of the SPR image into a probability density function of the geographic location associated with these SPR images).

[0032] More generally, localization can involve adjustments to the predicted vehicle position on a map or to the map itself via the map update module 340, which alters the map from the map database 216 based on the localization estimate generated by the deep learning submodule 315. Health metrics can include estimated repair times, estimated failure times for individual components and the entire system, accuracy estimates, estimated damage levels for physical sensor components, estimates of external interference, and similar indicators of system performance, durability, and reliability. Metrics can take the form of confidence values ​​for the entire system and / or individual submodules, as well as estimated accuracy and error distributions and estimated system latency. Generated metrics can also facilitate automated system parameter adjustments, such as gain adjustments. These operations can be processed by the metrics module 345.

[0033] In one embodiment, the deep learning submodule 315 includes a CNN that analyzes incoming SPR images (e.g., periodically sampled from the GPR antenna array 110) and calculates a match probability with one or more registered images. Alternatively, the SPR images can be analyzed conventionally, as described in the '024 patent, to locate one or more best-matching images and associated match probabilities. The match probabilities can be adjusted based on input received from sensors that capture conditions relevant to the reliability of the position estimate and processed by the deep learning submodule 315. This data can be processed by a different submodule 315 (e.g., another neural network) and can include environmental conditions (e.g., wind speed and / or temperature), vehicle parameters (e.g., speed), and / or SPR sensor parameters (e.g., sensor health or other performance indicators)—any data relevant to the reliability of the generated SPR data scan and / or the match probability with the reference image, and therefore, the accuracy of the position fix, expressed as, for example, an error estimate. The relationship between the data and the SPR images can be very complex, making the most relevant features difficult to detect and use as the basis for position fix, which is why a deep learning system is employed. The deep learning submodule 315 takes the sensor data and the raw SPR image as input and outputs a predicted position or data that can be used by a position estimation module 347, which can estimate the position using image registration as described above and in the '024 patent.

[0034] The way the deep learning submodule is trained depends on its architecture, input data format, and objectives. Typically, a large amount of input data is collected and georeferenced using real information (i.e., known locations) 350. The positioning error of the system is evaluated using a cost function and fed back to adjust the system weights. The deep learning system can also be trained to identify underground features whose details or structures may vary. As a conceptual example, utility pipes can vary not only in diameter but also in their orientation relative to the SPR sensor. It may be impractical or impossible to analytically represent all possible SPR images corresponding to the pipe, but by training a CNN (or, in the case where recognition requires sequential analysis of multiple SPR images, by a recurrent neural network or RNN), conduits in arbitrary orientations can be identified with high accuracy. Therefore, the neural network can be used to identify and register the features it has been trained to identify along the underground route - associating these features with the latitude / longitude coordinates obtained as described above or using GPS to generate a map of fixed, permanent, or semi-permanent underground structures before road or infrastructure construction.

[0035] Alternatively, feature recognition can be used for navigation purposes. For example, knowing that a pipeline of a certain size is located at specific GPS coordinates may be sufficient to determine the position of a moving vehicle based on general detection of pipelines near known locations. Accuracy can be improved if multiple features are detected at known distances apart. Subsurface features can also represent hazards or indicate the need for preventative measures to avoid them. For example, a subsurface area with high water content under a roadway could result in a pothole. If detected and corrected before a freeze-thaw cycle, hazardous road conditions can be avoided and mitigation costs reduced.

[0036] In some embodiments, the deep learning module 315 is hosted locally on a computing device within the vehicle and can be updated from time to time by a server as further centralized training improves the performance of the neural network. In other embodiments, the latest neural network model can be stored and accessed remotely by the vehicle via a wireless connection (e.g., via the Internet). Map updates and maintenance can also be performed "in the cloud."

[0037] The deep learning module 315 can be implemented without undue experimentation using commonly available libraries. Caffe, CUDA, PyTorch, Theano, Keras, and TensorFlow are suitable neural network platforms (the implementation can be cloud-based or local, depending on design preference). The input to the neural network can be a vector of input values ​​(a "feature" vector), for example, SPR scan readings and system health information.

[0038] Controller 208 may include one or more modules implemented in hardware, software, or a combination of both. For embodiments in which functionality is provided as one or more software programs, the programs may be written in any of a number of high-level languages, such as Python, Fortran, Pascal, Java, C, C++, C#, Basic, various scripting languages, and / or HTML. Additionally, the software may be implemented in assembly language targeted to a microprocessor resident on the target computer; for example, if the software is configured to run on an IBM PC or PC clone, it may be implemented in Intel 80x86 assembly language. The software may be embodied on an article of manufacture, including but not limited to a floppy disk, a flash drive, a hard disk, an optical disk, a magnetic tape, a PROM, an EPROM, an EEPROM, a field programmable gate array, or a CD-ROM.

[0039] The CPU 330 that executes commands and instructions can be a general-purpose computer, but can also utilize any of a variety of other technologies, including special-purpose computers, microcomputers, microprocessors, microcontrollers, peripheral integrated circuit components, CSICs (customer-specific integrated circuits), ASICs (application-specific integrated circuits), logic circuits, digital signal processors, programmable logic devices such as FPGAs (field programmable gate arrays), PLDs (programmable logic devices), PLAs (programmable logic arrays), RFID processors, smart chips, or any other device or apparatus capable of implementing the steps of the process of the present invention.

[0040] The terms and expressions used herein are intended to be descriptive and not restrictive, and when used, there is no intention to exclude any equivalents of the features shown and described or portions thereof. In addition, while certain embodiments of the present invention have been described, it will be apparent to those skilled in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the present invention. The described embodiments should therefore be considered to be illustrative in all respects and not restrictive.

[0041] The claims are:

Claims

1. A method for detecting and identifying underground structures, the method comprising the following steps: Acquire multiple surface penetrating radar (SPR) images during the traverse route; Acquiring sensor data of conditions related to reliability of the position estimate, the acquired sensor data including SPR sensor parameters; identifying subsurface structures in the acquired SPR images by a processor executing electronically stored instructions using the acquired SPR images as input to a predictor that has been trained to identify subsurface structures in SPR images; associating the identified subsurface structure in the acquired SPR image with ground coordinates corresponding to the geographic location of the identified subsurface structure, and based thereon, generating a ground map corresponding to the identified subsurface structure; as well as The vehicle localization is estimated by using the acquired SPR image and the acquired sensor data as inputs to a deep learning module, wherein the deep learning module calculates a probability of matching the acquired SPR image with one or more registered images and adjusts the matching probability based on the acquired sensor data.

2. The method according to claim 1, wherein The predictor is a neural network.

3. The method according to claim 1, wherein The predictor is a convolutional neural network.

4. The method according to claim 1, wherein The predictor is a recurrent neural network.

5. The method according to claim 1, further comprising the steps of: acquiring additional SPR images during vehicle traversal of the route; identifying, by the predictor, in additional SPR images, subsurface features that the predictor has been trained to identify; as well as The vehicle is navigated based at least in part on the identified subsurface features and their ground coordinates in the ground map.

6. A system for detecting and identifying underground structures, the system comprising: a surface penetrating radar (SPR) system for acquiring an SPR image and sensor data related to a condition of reliability of a position estimate, the acquired sensor data including SPR sensor parameters; and A computer comprising a processor and electronically stored instructions, the instructions being executable by the processor to: (i) analyze an acquired SPR image and identify subsurface structures therein by using the acquired SPR image as input to a predictor, the predictor being trained to identify subsurface structures in the SPR image, by the processor executing the electronically stored instructions; (ii) associating the identified subsurface structure in the acquired SPR image with ground coordinates corresponding to the geographical location of the identified subsurface structure, and based on this, generating a ground map of the subsurface structure corresponding to the identified subsurface structure; and (iii) estimating the vehicle localization by providing the acquired SPR image and the acquired sensor data as inputs to a deep learning module, wherein the deep learning module calculates a probability of matching the acquired SPR image with one or more registration images and adjusts the matching probability based on the acquired sensor data.

7. The system according to claim 6, wherein: The predictor is a neural network.

8. The system according to claim 6, wherein: The predictor is a convolutional neural network.

9. The system according to claim 6, wherein: The predictor is a recurrent neural network.

10. A vehicle comprising: a surface penetrating radar (SPR) system for acquiring an SPR image and sensor data related to a condition of position estimation reliability while the vehicle is traveling, the acquired sensor data including SPR sensor parameters; and A computer comprising a processor and electronically stored instructions executable by the processor for: analyzing the acquired SPR image and identifying subsurface structures therein by using the acquired SPR image as input to a predictor via a processor executing electronically stored instructions, the predictor having been trained to identify subsurface structures in the SPR image; estimating the vehicle's localization by using the acquired SPR image and the acquired sensor data as inputs to a deep learning module, wherein the deep learning module calculates a probability of matching the acquired SPR image with one or more registered images and adjusts the probability of matching based on the acquired sensor data; as well as The vehicle is navigated based at least in part on the identified underground structure and a surface map associating the identified underground structure with surface coordinates.

11. The vehicle according to claim 10, wherein: The computer is configured to associate the identified underground structures in the image with ground coordinates corresponding to the time when the image was obtained, and based on this, generate an electronic map corresponding to the identified underground structures.

12. The vehicle according to claim 10, wherein: The computer is configured to associate features identified in the image with ground coordinates corresponding thereto and to navigate the vehicle based at least in part on the identified subsurface features and their ground coordinates.

Images