Road surface condition detection using distributed optical fiber systems

Abstract

Fiber optic sensing cables installed at the side of the pavement and parallel to the driving direction are monitored by distributed fiber optic sensing (DFOS) using Rayleigh backscattering generated along the optical sensor fiber cable under the dynamic load of the vehicle. Vehicle interaction with road locations exhibiting deteriorated pavement surfaces generates unique patterns of localized signals, and these signals are identified / distinguished from signals arising from vehicles traveling on roads exhibiting smooth pavement surfaces. Machine learning techniques are employed to estimate the overall road surface quality and locate the locations where the pavement has deteriorated. Power Spectral Density Estimation, Principal Component Analysis (PCA), Support Vector Machine (SVM) combined with Principal Component Analysis, Local Binary Patterns (LBP), and Convolutional Neural Networks (CNN) are applied to develop the machine learning models.

Description

[Technical Field]

[0001] This disclosure relates generally to distributed fiber optic sensing (DFOS) systems, methods, and structures, and more particularly to systems, methods, and structures for road surface condition detection using DFOS technology. [Background technology]

[0002] Road surface conditions can have a significant impact on vehicle interaction with the road / pavement structure, potentially resulting in increased fuel consumption, increased vehicle maintenance costs, and even driver safety hazards. Pavement maintenance and repair actions are often determined based on measured pavement deterioration and road surface irregularity.

[0003] Walking surveys and image-based equipment are widely used approaches to detect pavement deterioration. Rut measurements and interior profilers are used, for example, by state transportation departments to measure and assess pavement unevenness levels. However, these approaches require labor-intensive field testing and post-processing and are not compatible with continuous monitoring and communication using smart infrastructure. Early detection of abnormal road conditions in real time is necessary to prevent further pavement damage and reduce agency and user costs caused by uneven road conditions. Summary of the Invention

[0004] The technology is advanced by aspects of the present disclosure directed to distributed fiber optic sensing of roads, where a fiber optic sensing cable positioned along the side of a pavement and parallel to the driving direction is monitored by distributed fiber optic sensing (DFOS) using Rayleigh backscattering generated along the optical sensor fiber cable under dynamic vehicle loads. Vehicle interaction with road locations exhibiting deteriorated pavement surfaces generates unique localized signal patterns, which are identified / distinguished from signals resulting from a vehicle traveling on roads exhibiting smooth pavement surfaces. Machine learning techniques are employed to estimate the overall quality of the road surface and locate areas of pavement deterioration. Machine learning models are developed using power spectral density estimation, principal component analysis (PCA), support vector machines (SVM) combined with PCA, local binary patterns (LBP), and convolutional neural networks (CNN).

[0005] A more complete understanding of the present disclosure may be realized by reference to the accompanying drawings. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating an exemplary DFOS system according to an embodiment of the present disclosure.

[0007] [Figure 2] FIG. 2 is a flow diagram illustrating the overall operational flow of a method according to an embodiment of the present disclosure.

[0008] [Figure 3] FIG. 3 is a schematic diagram illustrating an exemplary system setup for automatic pavement unevenness level detection, according to an embodiment of the present disclosure.

[0009] [Figure 4]FIG. 4 is a schematic diagram illustrating an exemplary framework for analysis according to an embodiment of the present disclosure.

[0010] [Figure 5] FIG. 5 is a schematic diagram illustrating an exemplary experimental setup according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following merely illustrates the principles of the present disclosure, and it should thus be understood that those skilled in the art will be able to devise various arrangements which embody the principles of the present disclosure, even though not explicitly described or shown herein, and which are within the spirit and scope of the present disclosure.

[0012] Furthermore, all examples and conditional language provided herein are meant to be for educational purposes only to aid in understanding the principles of the present disclosure and concepts provided by the inventors to further the present technology, and should not be construed as being limited to the specifically listed examples and conditions.

[0013] Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Furthermore, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., elements developed that perform the same function, regardless of structure.

[0014] Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure.

[0015] Unless otherwise specified, the drawings herein, including the figures, are not drawn to scale.

[0016] As additional background, we first note that distributed fiber optic sensing (DFOS) is an important and widely used technology for detecting environmental conditions (temperature, vibration, acoustic excitation, strain levels, etc.) anywhere along a fiber optic cable, which is connected to an interrogator. As is well known, modern interrogators are systems that generate an input signal to the fiber and detect and analyze the reflected / scattered signal that is subsequently received. The signal is analyzed to generate an output that is indicative of the environmental conditions occurring along the fiber. The received signal can result from reflections within the fiber, such as Raman backscattering, Rayleigh backscattering, or Brillouin backscattering. DFOS can also use forward signals that exploit the velocity differences of multiple modes. Without loss of generality, the following description assumes a reflected signal, but the same approach can be applied to forward signals.

[0017] Figure 1 is a schematic diagram of a generalized DFOS system. As understood, a modern DFOS system includes an interrogator that periodically generates an optical pulse (or any coded signal) and launches it into an optical fiber. The launched optical pulse signal is transmitted along the optical fiber.

[0018] At locations along the fiber, a small portion of the signal is reflected back to the interrogator. The reflected signal carries information that the interrogator uses for detection, such as changes in power level indicative of mechanical vibrations. As will be understood and appreciated, the interrogator may include a coded DFOS system employing a coherent receiver configuration known in the art.

[0019] The reflected signal is converted to the electrical domain and processed by an interrogator. Based on the time the pulse arrived and the time the signal was detected, the interrogator can determine from which point in the fiber the signal is coming and sense activity at each point in the fiber.

[0020] Those skilled in the art will understand and appreciate that implementing signal coding on the interrogation signal can advantageously improve the signal-to-noise ratio (SNR) of Rayleigh scattering-based systems (e.g., distributed acoustic sensing, or DAS) and Brillouin scattering-based systems (e.g., Brillouin optical time-domain reflectometry, or BOTDR) by allowing more optical power to be transmitted into the fiber.

[0021] As currently implemented in many modern implementations, DFOS systems in fiber optic cables are allocated dedicated fibers and are physically separated from existing optical communication signals carried in different fibers. However, given the exponential growth in bandwidth demand, it is becoming increasingly difficult to economically operate and maintain optical fiber solely for DFOS operations. As a result, there is growing interest in integrating communication and sensing systems onto a common fiber that is part of a larger multi-fiber cable.

[0022] Operationally, we envision DFOS systems to be Rayleigh scattering-based systems (e.g., distributed acoustic sensing or DAS) and Brillouin scattering-based systems (e.g., Brillouin optical time-domain reflectometry or BOTDR) with coding implementations. Such coding designs make these systems more likely to be integrated with fiber communication systems as they operate at lower power and are less sensitive to the response time of optical amplifiers.

[0023] Previously, we described how the DFOS system can be used to monitor road conditions by analyzing sensing data in the frequency domain and vehicle trajectories. Now, we leverage spatiotemporal data from the DFOS system to enhance these methods, enabling automated and continuous monitoring of the quality of in-service pavement surfaces, detecting significant deterioration (e.g., potholes and cracks), and sending warning messages to operators for rapid response and repair.

[0024] As those skilled in the art will understand and appreciate, such an approach requires solutions to several difficult problems: in order to detect signals from vehicles traveling on degraded, poor quality road surfaces as well as good quality road surfaces, the DOFS system must be able to identify signal features that are unique when a vehicle is traveling on a degraded road, i.e., a road with holes, cracks, etc., as well as when the vehicle is traveling on a normal, undegraded road surface.

[0025] Thus, systems, methods, and structures according to aspects of the present disclosure include at least two key features: estimating overall road surface quality and identifying spots or roads that exhibit significant degradation. An event algorithm is described for this second feature. An event is an abnormal driving signal generated when a vehicle drives over a defective portion of the road surface.

[0026] 2 is a flow diagram illustrating the overall operational flow of a method according to an embodiment of the present disclosure. As shown in this figure, the system and method of the present invention according to an embodiment of the present disclosure includes an installation of fiber optic sensor cables arranged parallel to a highway / roadway. A DFOS system is operatively connected to the installed fiber optic sensor cables to generate a waterfall plot of infrastructure vibrations as vehicles pass by. A machine learning model evaluates data obtained from the DFOS driving operation and evaluates the roughness level of the road surface as "good," "average," "poor," etc. Based on such data, the machine learning model detects abnormal locations on the road.

[0027] 3 is a schematic diagram illustrating an exemplary system configuration for automatic pavement unevenness level detection according to an embodiment of the present invention. As can be seen from the above, it can be divided into two parts: deployment of a sensing system and data analysis.

[0028] Step 1: Deploy the system

[0029] DFOS systems are installed in remote terminals / central offices and provide real-time, long-term, continuous remote monitoring. DFOS systems may be configured to operate as distributed acoustic sensors (DAS) or distributed vibration sensors (DVS).

[0030] The DFOS system is operatively connected to an optical sensor fiber, which may advantageously be an existing deployed telecommunications fiber or a newly deployed telecommunications fiber, such sensor fiber being capable of simultaneously carrying live communications traffic in addition to the DFOS probe and scattering signals.

[0031] FIG. 4 is a schematic diagram illustrating an exemplary framework for analysis according to an embodiment of the present invention.

[0032] Step 2: Data analysis

[0033] The illustrated analysis framework shows the first steps of collecting and annotating the data, then using a Hough transform, vehicle speed is estimated based on the detected lines.

[0034] Two methods are employed to detect potholes and cracks in road surfaces. The first method uses local binary pattern histograms to extract features from annotated images. Principal component analysis (PCA) is employed to reduce the dimensionality, and support vector machine (SVM) is considered as a classifier to train the model.

[0035] The second method uses a convolutional neural network (CNN) to train on binary classification data to obtain a trained model. Training begins with image augmentation.

[0036] Finally, a model trained based on SVM and CNN methods is deployed to detect speed bumps on the pavement and the level of pavement roughness.

[0037] Hough transform

[0038] The Hough transform is a general feature extraction technique for detecting any shape that can be expressed in mathematical form. It is effective for detecting bent or curved shapes, and lines can be expressed in terms of two parameters, a and b, as y = a·x + b. This formula cannot be used to represent perpendicular lines, so normal parameterization is commonly used. Normal parameterization defines a line by its normal angle θ and its algebraic distance from the origin ρ, as shown in the following equation:

number

number

[0039] Thus, the Hough space of a line has two dimensions, θ and ρ, and a line can be represented by a unique point (θ0, ρ0) in the θ-ρ plane.

[0040] A set of n figure points

number

number

number

number

[0041] Local Binary Pattern (LBP)

[0042] LBP is a type of visual descriptor that combines the properties of statistical texture analysis and structural texture analysis. LBP-based algorithms are widely used to detect cracks in road pavement by extracting edge orientation and texture features from camera-captured images. Advantages of LBP include high discriminatory power, computational simplicity, and invariance to grayscale changes. This is performed based on grayscale-invariant 2D texture analysis. In LBP, image pixels are labeled by thresholding each pixel's neighborhood by a median value and considering the result of this thresholding as a binary number.

[0043] For example, if the luminance of the central pixel is less than the luminance of the sample pixel, it is set to a value of "1", otherwise it is set to a value of "0". A new matrix contains these binary values, and the central value is ignored. The binary values ​​of each location are aggregated row by row from the matrix to a new binary value. The binary value is converted to a decimal value and set as the central value of the matrix. The number of cells is used to divide the image into multiple grids and a histogram of the features is plotted.

[0044] Officially,

number

number

number

[0045] From the above definition, the basic LBP operator is invariant to monotonic grayscale transformations that preserve the order of pixel intensities in a local neighborhood. The histogram of LBP labels computed over a region can be used as a texture descriptor.

[0046] Support Vector Machine (SVM)

[0047] SVM generates nonlinear boundaries by constructing large linear boundaries in a transformed version of the feature space. The basic principle of SVM is to find an optimal separating hyperplane that creates the maximum margin between the training data. The training points closest to the hyperplane are called support vectors

[25] . The training data are

number

number

number

[0048] where β is a unit vector

number

number

[0049] where f(x) is the hyperplane from point x

number

[0050] The classes are separated, so

number

number

[0051] Optimization Problem

number

number

number

number

number

[0052] Convolutional Neural Networks (CNNs)

[0053] CNNs are particularly applied to image classification for structural health monitoring. This network topology focuses on the ability of a computer system to learn patterns in data without the patterns being explicitly programmed. CNNs can be implemented in Keras to improve classification accuracy using backends such as Theano and Tensorflow. Keras is a high-level neural network written in Python.

[0054] CNN is used to reduce the actual input size of an image by performing convolution, identifying important image features required for classification, and ignoring unnecessary features. The CNN architecture consists of a convolutional layer, a max pooling layer, and a classification layer. The convolutional layer is a layer composed of multiple filters, each of which is convolved with the input image according to a program. Depending on the number of features applied, the input shape of the image is reduced by the convolutional layer. A feature map of the image is obtained. The table below shows the details of the convolutional network used to train the model in this study. [Table 1]

[0055] The Keras neural network library can perform image augmentation, suitable for small datasets, to prevent overfitting. Image augmentation is a method of transforming input images using specific parameters, teaching the network that all images are the same regardless of the transformation performed. There are many available image augmentation types, and the image augmentation type must be selected taking into account the meaning of the modified image. In this study, width shift, height shift, and horizontal flip are selected as parameters to generate more data. Keras has a built-in process and is performed by a class called an image data generator.

[0056] Data collection and preparation

[0057] FIG. 5 is a schematic diagram illustrating an exemplary experimental setup according to an embodiment of the present disclosure.

[0058] This figure shows an exemplary experimental setup with a DAS system installed in a central office (CO) and speed bumps installed on the road surface. When a vehicle passes over a speed bump, the vibration signals generated effectively simulate road anomalies such as potholes or large cracks. The speed bumps employed are 6-inch-tall rubber safety bumps, positioned perpendicular to the optical fiber.

[0059] The waterfall data collected from the experimental runs is subjected to a Hough transform. From the transformed data, vehicle vibrations, speed bump characteristics, and vehicle speed can be determined. The Hough transform facilitates localization of vehicle passing events in the spatiotemporal plane of the waterfall and is also useful for detecting the location of anomalous driving signals, since signals indicative of potholes and cracks always appear along the vehicle's trajectory. The subsequent classifier scans the vehicle and adjacent image patches when detecting pothole locations, rather than the entire waterfall plot. Advantageously, waterfall signals are very sparse, with only a small amount of data showing useful information, significantly reducing computational costs.

[0060] Furthermore, our experimental observations yielded waterfall traces over a 100-second window containing 400 sensing points. Because there was only one speed bump installed on the road, two strips were observed in the plot, with each strip representing one axle of the vehicle passing over the speed bump. As expected, when the vehicle was traveling over the bump, ground vibrations increased dramatically compared to normal road vibrations. The intensity at the speed bump location was higher than that at the non-bump location.

[0061] Further tests were conducted during rainy weather to obtain waterfall traces under wet road conditions. Ground vibrations were observed to weaken after rainfall, which may have resulted in changes in the mechanical interaction between the road surface and vehicle tires, the elastic modulus of the pavement structure, and the elastic modulus of the soil.

[0062] As is well known, DAS is a relatively recently developed technology that uses fiber optic cables to measure ground motion. DAS systems can provide distributed strain sensing based on Rayleigh scattering. Dynamic strain changes along the optical fiber can be captured and correlated with the surrounding acoustic field. DAS has several advantages over geophones. Unlike geophones, DAS cables are thin, so there are no limitations in horizontal or ultra-slim tubes. Fiber optics are low cost and easy to install with other fiber optic sensors, including distributed temperature sensing (DTS) and distributed pressure sensing (DPS).

[0063] "DAS can measure across the entire horizontal range without moving the fiber, ensuring data continuity. DAS systems can detect dynamic vibrations across the entire optical fiber with high spatial resolution. Rather than deploying new DFOS or strain gauges in the pavement structure, existing commercial communications equipment and tools can be used to monitor traffic and road surface conditions."

[0064] The collected images are classified into two main classes: with bumps and without bumps. If the road has speed bumps, images are taken at the corresponding locations. To ensure that the sample size is large enough, images are randomly sampled from the waterfall plot of the normal road surface class without bumps. The size of these images is 24x30 pixels in PNG (Portable Network Graphics) format.

[0065] Two datasets were collected in central New Jersey. Field tests were conducted on June 25, 2019, and August 7, 2019, respectively. The June 25th test was sunny and the road surface was dry. 250 images were extracted for the dataset, including pavement sections with and without speed bumps. The August 7th test was rainy, and the road surface and soil were wet. 120 images were extracted for the dataset, including both cases. Because the weather conditions varied between the two tests, the datasets were processed and analyzed separately. All data processing and analysis was performed in Python with the necessary modules.

[0066] Hough transform based line detection

[0067] A waterfall plot of the data utilized a Hough transform to detect straight lines by connecting points or strips of high intensity generated by bumps on the pavement. The slope of the line indicates the vehicle speed. The number of vehicles can also be counted by integrating the number of straight lines. The estimated speeds were 5.6 m / s (20 km / h) and 6.4 m / s (23 km / h), respectively. Due to the concrete structure, an increase in the vibration signal was observed at the locations of puddles between the bumps.

[0068] Results of LBP and SVM-based methods

[0069] The dataset was divided into two sets: training data (90%) and test data (10%). LBP was applied to the data images, and histograms of the resulting images were obtained. Each histogram contained 256 positions indicating the presence or absence of each pixel's brightness, representing the characteristics of the original image. The results showed that the LBP histograms obtained on pavement sections with speed bumps were different from those obtained on pavement sections without speed bumps. The LBP histograms of each image can be considered as features for the classifier. Comparing the LBP histograms collected on dry roads with and without speed bumps revealed significant differences in the proportion of pixels between the two cases. For example, when the pixel brightness was between 250 and 255, the proportion of pixels without speed bumps was significantly higher than that with bumps. On the other hand, when the pixel brightness was between 145 and 155, the proportion of pixels with bumps was significantly higher. A similar trend was observed on wet roads.

[0070] The road surface condition (dry / wet) has a significant effect on the LBP histogram, which is effective in extracting image features for detecting road anomalies.

[0071] A confusion matrix of the classification results was created for the dataset collected on dry roads. As a preliminary step, predicted classes are represented by the rows of the matrix, and actual classes are represented by the columns of the matrix. Colors can be used to distinguish each region, corresponding to the number of predicted or predefined images in each class. The number of false positives was 0, and the number of false negatives was 2. The accuracy of the prediction model was 97%. The confusion matrix of the classification results for the dataset collected on wet roads showed that the number of false positives was 3, and the number of false negatives was 0. The accuracy of the prediction model was 90%. This comparison showed that weather and road surface conditions are important parameters for developing a prediction model.

[0072] Convolutional neural network results

[0073] For the convolutional neural network method, each dataset was divided into three sets: training (72%), validation (18%), and test data (10%). Training of the neural network for classification was performed for five epochs, and the model and weights were saved for future development and modification. Accuracy and loss curves were plotted. The accuracy for the training and validation data on dry roads was 100%, with a loss of 0.48. The accuracy for the training and validation data on wet roads was 100%, with a loss of 0.16.

[0074] The comparison of classification accuracy between the proposed methods shows that the convolutional network model is highly accurate for both dry and wet road conditions. However, the LBP histogram contains only limited structural information, and the computational complexity can increase depending on the feature size from a spatiotemporal perspective. However, the convolutional network training model required a large dataset to avoid the overfitting problem.

[0075] In conclusion, DFOS is implemented to capture ground vibrations under sunny and rainy conditions. Speed ​​bumps are installed on the road surface to reproduce the vibration signals generated by road anomalies and identify vehicle locations on multi-lane roads. LBP histograms can effectively characterize image features. LBP and SVM-based methods are effective in detecting road anomalies. Compared with LBP and SVM-based methods, CNN methods improve the accuracy of predictive models. However, a larger dataset is required to overcome the overfitting problem. DFOS is practical for traffic monitoring (traffic accumulation and vehicle speed estimation) and road anomaly detection. It has the potential to be implemented in smart roads. Because experiments could only be conducted at specific locations, the sample size was relatively small. Important parameters, such as pavement structure and environmental factors, cannot be included in models trained based on limited data. Collecting more datasets will improve accuracy and expand applications. In future research, we plan to utilize the trained model in a real-time engine for vehicle detection on multi-lane roads. This technique can also be extended to road anomaly detection by combining it with sliding window techniques.

[0076] While the present disclosure has been illustrated herein using certain specific examples, those skilled in the art will recognize that the present teachings are not limited thereto. Accordingly, the present disclosure should be limited only by the scope of the claims appended hereto.

Claims

1. 1. A method for detecting road surface conditions using distributed fiber optic sensing (DFOS), comprising: an optical sensor fiber disposed substantially parallel to the road surface; a DFOS interrogator in optical communication with the optical sensor fiber configured to generate optical pulses, input the generated pulses into the optical sensor fiber, and receive backscattered signals from the optical sensor fiber; an intelligent analyzer configured to analyze the backscattered signals received by the DFOS interrogator and identify vibrational activity occurring at locations along the optical sensor fiber from the backscattered signals; Equipped with determining the speed of a vehicle traveling on a road from the backscattered signals using a Hough transform; Identifying a location of a pothole or crack in the road surface and outputting information about the location of the pothole or crack; the intelligent analyzer generates an annotated waterfall plot image from the backscattered signal, annotated for machine learning, and determines the vibration activity occurring at locations along the optical sensor fiber by extracting features from the annotated waterfall plot image using a local binary pattern histogram, and by performing principal component analysis (PCA) on the annotated waterfall plot image to reduce its dimensionality, and generating a trained model using a support vector machine method as a classifier; A method for training binary classification data using a convolutional neural network (CNN) to generate a trained model.

2. The method of claim 1, further comprising: deploying a trained model based on a support vector machine (SVM) and a CNN to detect road roughness levels and classify the levels as good, average, or bad.

Images