Road condition detection using distributed optical fiber system

A distributed fiber optic sensing system with machine learning techniques addresses the challenge of labor-intensive road condition detection, enabling real-time monitoring and early alerting of pavement deterioration using existing telecommunications fibers.

JP2024544045AActive Publication Date: 2024-11-27NEC CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024528490
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-11-15
Filing Date
2022-11-16
Publication Date
2024-11-27
Estimated Expiration
2042-11-16

AI Technical Summary

Technical Problem

Existing methods for detecting road surface conditions, such as gait surveys and image-based instruments, are labor-intensive and not suitable for continuous monitoring, failing to provide real-time detection of abnormal road conditions, which can lead to increased maintenance costs and safety issues.

Method used

A distributed fiber optic sensing system using Rayleigh backscatter and machine learning techniques to analyze vehicle interactions with the road surface, employing power spectral density estimation, principal component analysis, support vector machines, local binary patterns, and convolutional neural networks to identify pavement deterioration.

Benefits of technology

Enables real-time, continuous monitoring of road surface quality, accurately detecting deteriorated areas and providing early alerts for maintenance, reducing costs and improving safety by leveraging existing telecommunications fibers for both communication and sensing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544045000001_ABST
    Figure 2024544045000001_ABST
Patent Text Reader

Abstract

Fiber optic sensing cables installed at the side of the pavement and parallel to the driving direction are monitored by distributed fiber optic sensing (DFOS) using Rayleigh backscattering generated along the optical sensor fiber cable under the dynamic load of the vehicle. Vehicle interaction with road locations exhibiting deteriorated pavement surfaces generates unique patterns of localized signals, and these signals are identified / distinguished from signals arising from vehicles traveling on roads exhibiting smooth pavement surfaces. Machine learning techniques are employed to estimate the overall road surface quality and locate the locations where the pavement has deteriorated. Power Spectral Density Estimation, Principal Component Analysis (PCA), Support Vector Machine (SVM) combined with Principal Component Analysis, Local Binary Patterns (LBP), and Convolutional Neural Networks (CNN) are applied to develop the machine learning models.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This disclosure relates generally to distributed fiber optic sensing (DFOS) systems, methods and structures, and more particularly to systems, methods and structures for road surface condition detection using DFOS techniques. [Background technology]

[0002] Road surface conditions can have a significant impact on vehicle interaction with the road / pavement structure, potentially resulting in increased fuel consumption, increased vehicle maintenance costs, and even driver safety hazards. Pavement maintenance and repair actions are often determined based on measured pavement deterioration and road surface irregularity conditions.

[0003] Walking surveys and image-based instruments are widely used approaches to detect pavement deterioration. Rut measurements and interior profilers are used, for example, by state transportation departments to measure and evaluate pavement unevenness levels. However, these approaches require labor-intensive field work testing and post-processing, and are not compatible with continuous monitoring and communication using smart infrastructure. Early detection of abnormal road conditions in real time is necessary to prevent further pavement damage and reduce agency and user costs incurred by uneven road conditions. Summary of the Invention

[0004] The advancement of the art is made by aspects of the present disclosure directed to distributed fiber optic sensing of roads where a fiber optic sensing cable placed along the pavement side and parallel to the driving direction is monitored by distributed fiber optic sensing (DFOS) using Rayleigh backscattering generated along the optical sensor fiber cable under dynamic vehicle loads. Vehicle interaction with locations on the roadway exhibiting deteriorated pavement surfaces generates unique patterns of localized signals that are identified / distinguished from signals resulting from vehicles traveling on roadways exhibiting smooth pavement surfaces. Machine learning techniques are employed to estimate the overall quality of the road surface and locate locations of pavement deterioration. Power spectral density estimation, principal component analysis (PCA), support vector machine (SVM) combined with principal component analysis, local binary pattern (LBP), and convolutional neural network (CNN) are applied to develop the machine learning model.

[0005] A more complete understanding of the present disclosure may be realized with reference to the accompanying drawings. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating an exemplary DFOS system according to an embodiment of the present disclosure.

[0007] [Diagram 2] FIG. 2 is a flow diagram illustrating the overall operational flow of a method according to an embodiment of the present disclosure.

[0008] [Diagram 3] FIG. 3 is a schematic diagram illustrating an example system setup for automatic pavement unevenness level detection according to an embodiment of the present disclosure.

[0009] [Figure 4]FIG. 4 is a schematic diagram illustrating an exemplary framework for analysis according to an embodiment of the present disclosure.

[0010] [Diagram 5] FIG. 5 is a schematic diagram illustrating an exemplary experimental setup according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following merely illustrates the principles of the present disclosure, and it should thus be understood that those skilled in the art can devise various configurations which embody the principles of the present disclosure, even if not explicitly described or shown herein, and which are within the spirit and scope of the present disclosure.

[0012] Furthermore, all examples and conditional terms given herein are meant for educational purposes only to aid in the understanding of the principles of the present disclosure and concepts provided by the inventors to further the present technology, and should not be construed as being limited to the specifically given examples and conditions.

[0013] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Moreover, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., elements developed that perform the same function, regardless of structure.

[0014] Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein are conceptual views illustrating illustrative circuitry embodying the principles of the disclosure.

[0015] Unless otherwise specified, the drawings herein, including the figures, are not drawn to scale.

[0016] As additional background, we first point out that Distributed Fiber Optic Sensing (DFOS) is an important and widely used technology for detecting environmental conditions (temperature, vibration, acoustic excitation vibration, strain levels, etc.) anywhere along a fiber optic cable, which is connected to an interrogator. As is well known, a modern interrogator is a system that generates an input signal to the fiber and detects / analyzes the reflected / scattered signal that is then received. The signal is analyzed and an output is generated that is indicative of the environmental conditions occurring along the fiber. The received signal may result from reflections within the fiber, such as Raman backscattering, Rayleigh backscattering, Brillouin backscattering, etc. DFOS can also use forward signals that exploit the velocity differences of multiple modes. Without loss of generality, the following description assumes reflected signals, but the same approach can be applied to forward signals.

[0017] A generalized DFOS system is shown in Figure 1. As understood, a modern DFOS system includes an interrogator that periodically generates a light pulse (or any coded signal) and launches it into an optical fiber. The launched light pulse signal is then transmitted along the optical fiber.

[0018] At locations along the fiber, a small portion of the signal is reflected back to the interrogator. The reflected signal carries information that the interrogator uses to detect, such as changes in power levels indicative of mechanical vibrations. As will be understood and appreciated, the interrogator may include a coded DFOS system employing coherent receiver configurations known in the art.

[0019] The reflected signal is converted to the electrical domain and processed by the interrogator, and based on the time the pulse arrives and the time the signal is detected, the interrogator can determine from which point in the fiber the signal is coming and sense activity at each point in the fiber.

[0020] Those skilled in the art will understand and appreciate that implementing signal coding on the interrogation signal can advantageously improve the signal-to-noise ratio (SNR) of Rayleigh scattering-based systems (e.g., distributed acoustic sensing, or DAS) and Brillouin scattering-based systems (e.g., Brillouin optical time domain reflectometry, or BOTDR) by allowing more optical power to be transmitted into the fiber.

[0021] As currently implemented in many modern implementations, DFOS systems in fiber optic cables are allocated dedicated fibers and are physically separated from existing optical communication signals carried in different fibers. However, given the exponential growth in bandwidth demand, it is becoming increasingly difficult to economically operate and maintain optical fiber just for DFOS operations. As a result, there is growing interest in integrating communication and sensing systems onto a common fiber that is part of a larger multi-fiber cable.

[0022] Operationally, we envision DFOS systems to be Rayleigh scattering-based systems (e.g., distributed acoustic sensing or DAS) and Brillouin scattering-based systems (e.g., Brillouin optical time-domain reflectometry or BOTDR) with coding implementations. With such coding designs, these systems have a high potential to be integrated with fiber communication systems as they operate at low power and are sensitive to the response time of optical amplifiers.

[0023] Previously, we have described how the DFOS system can be used to monitor road conditions by analyzing sensing data in the frequency domain and vehicle trajectories. Now, we are leveraging spatio-temporal data from the DFOS system to enhance these methods to automatically and continuously monitor the quality of in-service pavement surfaces, detect major deterioration (e.g. holes and cracks), and send warning messages to operators for rapid response and repair.

[0024] As those skilled in the art will understand and appreciate, such an approach requires several solutions to a difficult problem: in order to detect signals from vehicles traveling on degraded, poor quality road surfaces as well as good quality road surfaces, the DOFS system needs to identify signal features that are unique when a vehicle travels on a degraded road, i.e., a road with holes, cracks, etc., as well as when the vehicle travels on a normal, undegraded road surface.

[0025] Thus, the systems, methods and structures according to the embodiments of the present disclosure include at least two important features: estimating the overall road surface quality and identifying the locations of spots or roads that exhibit significant deterioration. An event algorithm is described for this second feature, where an event is an abnormal driving signal generated when a vehicle drives over a defective portion of the road surface.

[0026] FIG. 2 is a flow diagram illustrating the overall operational flow of a method according to an embodiment of the present disclosure. As shown in this figure, the system and method of the present disclosure according to an embodiment of the present disclosure includes an installation of fiber optic sensor cables arranged parallel to a highway / roadway. A DFOS system is operatively connected to the installed fiber optic sensor cables, and a waterfall plot of infrastructure vibrations is generated while a vehicle is passing through. A machine learning model evaluates data obtained from the DFOS driving operation and evaluates the roughness level of the road surface as "good", "average", "poor", etc. The machine learning model detects anomalous locations on the road based on such data.

[0027] 3 is a schematic diagram showing an exemplary system configuration for automatic pavement unevenness level detection according to an embodiment of the present invention. As can be seen from the above, it can be divided into two parts: deployment of a sensing system and data analysis.

[0028] Step 1: Deploy the system

[0029] DFOS systems are installed in remote terminals / central offices and provide real-time, long-term, continuous, remote monitoring. DFOS systems may be configured to operate as a distributed acoustic sensor (DAS) or a distributed vibration sensor (DVS).

[0030] The DFOS system is operatively connected to an optical sensor fiber, which may advantageously be an existing deployed telecommunications fiber or a newly deployed telecommunications fiber, such sensor fiber being capable of simultaneously carrying live communications traffic in addition to the DFOS probe and scattering signals.

[0031] FIG. 4 is a schematic diagram illustrating an exemplary framework for analysis according to an embodiment of the present invention.

[0032] Step 2: Data analysis

[0033] The illustrated analysis framework shows the first steps of collecting and annotating the data, then the vehicle speed is estimated based on the detected lines using the Hough transform.

[0034] Two methods are adopted to detect potholes and cracks in the road surface. The first method uses local binary pattern histograms to extract features from the annotated images. Principal Component Analysis (PCA) is employed to reduce the dimensionality, and Support Vector Machine (SVM) is considered as a classifier to train the model.

[0035] The second method uses a convolutional neural network (CNN) to train on binary classification data to obtain a trained model. Training starts with image augmentation.

[0036] Finally, a model trained based on SVM and CNN methods is deployed to detect speed bumps on the pavement and the pavement roughness level.

[0037] Hough Transform

[0038] The Hough transform is a general feature extraction technique for detecting any shape that can be expressed in mathematical form. It is effective at detecting bent or curved shapes, and lines can be expressed by two parameters, a and b, as y=a·x+b. This formula cannot be used to represent perpendicular lines, so normal parameterization is commonly used. Normal parameterization specifies a line by its normal angle θ and its algebraic distance from the origin ρ, as shown in the following formula:

number

number

[0039] Thus, the Hough space of a line has two dimensions, θ and ρ, and a line can be represented by a unique point (θ0, ρ0) in the θ-ρ plane.

[0040] A set of n points

number

number

number

number

[0041] Local Binary Pattern (LBP)

[0042] LBP is a type of visual descriptor that combines the properties of statistical and structural texture analysis. LBP-based algorithms are widely used to detect cracks in road pavement by extracting edge orientation and texture features from images captured by a camera. The advantages of LBP include high discriminatory power, computational simplicity, and invariance to grayscale changes. It is performed based on grayscale-invariant 2D texture analysis. In LBP, pixels in an image are labeled by thresholding each pixel's neighborhood with the median value and considering the result of this thresholding as a binary number.

[0043] For example, if the luminance of the central pixel is less than the luminance of the sample pixel, it is set to value "1", otherwise it is set to value "0". A new matrix contains these binary values ​​and the central value is ignored. The binary values ​​of each location are aggregated row by row from the matrix to a new binary value. The binary value is converted to a decimal value and set as the central value of the matrix. The number of cells is used to divide the image into multiple grids and a histogram of the features is plotted.

[0044] Officially,

number

number

number

[0045] From the above definition, the basic LBP operator is invariant to monotonic grayscale transformations that preserve the order of pixel intensities in a local neighborhood. A histogram of LBP labels computed over a region can be used as a texture descriptor.

[0046] Support Vector Machine (SVM)

[0047] SVM generates nonlinear boundaries by constructing large linear boundaries in a transformed version of the feature space. The basic principle of SVM is to find an optimal separating hyperplane that creates the maximum margin between the training data. The training points closest to the hyperplane are called support vectors

[25] . The training data are

number

number

number

[0048] where β is a unit vector

number

number

[0049] Here, f(x) is the hyperplane

number

[0050] The classes are separated,

number

number

[0051] Optimization Problem

number

number

number

number

number

[0052] Convolutional Neural Networks (CNNs)

[0053] CNNs are particularly applied in image classification for structural health monitoring. This network topology focuses on the ability of a computer system to learn patterns in data without the patterns being explicitly programmed. CNNs can be implemented in Keras to improve classification accuracy with backends such as Theano and Tensorflow. Keras is a high-level neural network written in Python.

[0054] CNN is used to reduce the actual input size of an image by performing convolution, revealing important features of the image required for classification, and ignoring unnecessary features. The CNN architecture consists of a convolution layer, a max pooling layer, and a classification layer. The convolution layer is a layer that consists of multiple filters, and each filter is convolved with the input image according to a program. Depending on the number of features applied, the input shape of the image is reduced by the convolution layer. A feature map of the image is obtained. The table below shows the details of the convolution network used to train the model in this study. [Table 1]

[0055] The Keras neural network library can perform image augmentation, which is suitable for small datasets, to prevent the overfitting problem. Image augmentation is a method to transform input images with certain parameters and teach the network that all images are the same regardless of the transformation performed. There are many image augmentation types available, and the image augmentation type should be selected by considering the meaning of the modified image. In this study, width shift, height shift, and horizontal flip are selected as parameters to generate more data. Keras has this process built in and is performed by a class called image data generator.

[0056] Data collection and preparation

[0057] FIG. 5 is a schematic diagram illustrating an exemplary experimental setup according to an embodiment of the present disclosure.

[0058] This figure shows an exemplary experimental setup with a DAS system installed in a central office (CO) and speed bumps installed on the road surface. When a vehicle passes over the speed bump, the vibration signals generated effectively simulate road anomalies such as potholes and large cracks. The speed bumps employed are 6-inch high rubber safety bumps, positioned perpendicular to the optical fiber.

[0059] The waterfall data collected from the experimental runs are subjected to a Hough transform. From the transformed data, vehicle vibrations, speed bump characteristics and vehicle speed can be determined. The Hough transform facilitates localization of vehicle passing events in the space-time plane of the waterfall and is also useful for detecting the location of anomalous driving signals, since signals indicative of potholes and cracks always appear on the vehicle's trajectory. The subsequent classifier scans the vehicle and adjacent image patches when detecting the pothole locations, rather than the entire waterfall plot. Advantageously, the waterfall signal is very sparse and only a small amount of data shows useful information, thus significantly reducing the computational cost.

[0060] Furthermore, our experimental observations yielded waterfall traces over a 100-second window containing 400 sensing points. Since there was only one speed bump installed on the road, two strips are observed in the plot, with each strip representing one axle of the vehicle passing over the speed bump. As expected, when the vehicle was traveling over the bump, the ground vibrations increased dramatically compared to the normal road vibrations. The intensity at the speed bump location was higher than the intensity at the non-bump location.

[0061] Further tests were conducted in rainy weather to obtain waterfall traces under wet road conditions. Ground vibrations were observed to weaken after rainfall, which may have caused changes in the mechanical interaction between the road surface and vehicle tires, the elastic modulus of the pavement structure, and the elastic modulus of the soil.

[0062] As is well known, DAS is a relatively recently developed technology that uses fiber optic cables to measure ground motion. DAS systems can provide distributed strain sensing based on Rayleigh scattering. Dynamic strain changes along the optical fiber can be captured and related to the surrounding acoustic field. DAS has several advantages over geophones. Unlike geophones, DAS has a thin cable and is not limited by horizontal or ultra-slim tubes. Fiber optics are low cost and easy to install with other fiber optic sensors, including distributed temperature sensing (DTS) and distributed pressure sensing (DPS).

[0063] DAS can measure over the entire horizontal range without moving the fiber, ensuring data continuity. The DAS system can detect dynamic vibrations over the entire optical fiber with high spatial resolution. Rather than deploying new DFOS or strain gauges in the pavement structure, existing commercial communication equipment and tools can be used to monitor traffic and road conditions.

[0064] The collected images are classified into two main classes: with bumps and without bumps. In cases with speed bumps, images are taken at the corresponding locations. To ensure that the sample size is large enough, images are randomly drawn from the waterfall plot of the normal road surface class without bumps. The size of these images is 24x30 pixels in PNG (Portable Network Graphics) format.

[0065] Two datasets were collected in central New Jersey. The field tests were conducted on June 25, 2019 and August 7, 2019, respectively. On June 25, the weather was sunny and the road surface was dry. For the dataset, 250 images were extracted, including pavement sections with and without speed bumps. On August 7, 2019, it was raining, and the road surface and soil were wet. For the dataset, 120 images were extracted, including both the above cases. As the weather conditions changed during the two tests, the datasets are processed and analyzed separately. All data processing and analysis are performed in Python with the necessary modules.

[0066] Hough transform based line detection

[0067] A waterfall plot of the data utilized a Hough transform to detect straight lines by connecting points or strips of high intensity generated by bumps on the pavement. The slope of the line indicates the vehicle speed. The number of vehicles can also be counted by integrating the number of straight lines. The estimated speeds were 5.6 m / s (20 km / h) and 6.4 m / s (23 km / h), respectively. Due to the concrete construction, an enhancement of the vibration signal was observed at the locations of the puddles between the bumps.

[0068] Results of LBP and SVM-based methods

[0069] The dataset is split into two sets: training data (90%) and testing data (10%). LBP is applied to the images of the data and histograms of the resulting images are obtained. Each histogram contains 256 positions that indicate the presence of brightness for each pixel, and the histogram represents the characteristics of the original image. The results show that the LBP histograms obtained on pavement sections with speed bumps are different from those obtained on pavement sections without speed bumps. The LBP histograms of each image can be considered as features in the classifier. Comparing the LBP histograms with and without speed bumps collected on dry roads, it can be seen that there is a significant difference in the proportion of pixels between the two cases. For example, when the pixel brightness is between 250 and 255, the proportion of pixels without bumps is obviously larger than that with bumps. On the other hand, when the pixel brightness is between 145 and 155, the proportion of pixels with bumps is larger. A similar trend was observed on wet roads.

[0070] Road surface condition (dry / wet) has a significant impact on the LBP histogram, which was effective in extracting image features for detecting road anomalies.

[0071] A confusion matrix of the classification results was created for the dataset collected on dry road surfaces. As a preliminary, the predicted classes are represented by the rows of the matrix, and the actual classes are represented by the columns of the matrix. Colors can be used to distinguish each region, corresponding to the number of predicted or predefined images in each class. The number of false positives was 0 and the number of false negatives was 2. The accuracy of the prediction model was 97%. The confusion matrix of the classification results for the dataset collected on wet road surfaces showed that the number of false positives was 3 and the number of false negatives was 0. The accuracy of the prediction model was 90%. This comparison showed that weather and road surface conditions are important parameters for the development of a prediction model.

[0072] Convolutional Neural Network Results

[0073] For the convolutional neural network method, each dataset was divided into three sets: training (72%), validation (18%), and test data (10%). Training of the neural network for classification was performed for 5 epochs, and the model and weights were stored for future development and modification. Accuracy and loss curves were plotted. The accuracy of the training and validation data on dry roads was 100%, and the loss was 0.48. The accuracy of the training and validation data on wet roads was 100%, and the loss was 0.16.

[0074] The classification accuracy comparison results between the proposed methods show that the convolutional network model has high accuracy in both dry and wet road conditions. The LBP histogram contains only limited structural information, and the computational complexity may increase with the feature size from a spatiotemporal perspective. However, the convolutional network training model required a large dataset to avoid the overfitting problem.

[0075] In conclusion, DFOS is implemented to capture ground vibrations under sunny and rainy weather. Speed ​​bumps are installed on the road surface to reproduce the vibration signals generated by road anomalies and to identify the location of vehicles on multi-lane roads. The LBP histogram can effectively characterize the image features. The LBP and SVM-based methods are effective in detecting road anomalies. Compared with the LBP and SVM-based methods, the CNN method improves the accuracy of the prediction model. However, more datasets are needed to overcome the overfitting problem. DFOS is practical for traffic monitoring (traffic accumulation and vehicle speed estimation) and road anomaly detection. It has the potential to be implemented in smart roads. The size of the sample was relatively small because the experiments could only be conducted at specific locations. Important parameters such as pavement structure and environmental factors cannot be included in the trained model based on limited data. If more datasets are collected, the accuracy will be improved and the application will be expanded. In future work, we plan to utilize the trained model in a real-time engine for vehicle detection on multi-lane roads. This technique can also be extended to road anomaly detection by combining it with sliding window techniques.

[0076] Although the present disclosure has been illustrated herein using certain specific examples, those skilled in the art will recognize that the present teachings are not so limited. Accordingly, the present disclosure should be limited only by the scope of the claims appended hereto.

Claims

1. 1. A method for detecting road surface conditions using distributed optical fiber sensing (DFOS), comprising: An optical sensor fiber disposed substantially parallel to the road surface; a DFOS interrogator in optical communication with the optical sensor fiber configured to generate optical pulses, input the generated pulses into the optical sensor fiber, and receive backscattered signals from the optical sensor fiber; an intelligent analyzer configured to analyze the DFOS data received by the DFOS interrogator and identify vibration activity occurring at locations along the optical sensor fiber from the backscattered signals; Equipped with determining a speed of a vehicle traveling on a road from the backscattered signals using a Hough transform; The method further comprises: locating holes / cracks in the road surface; and outputting information on the location of the holes / cracks.

2. The method of claim 1 , further comprising generating an annotated waterfall plot image from the backscattered signals and extracting features of the annotated image using a local binary pattern histogram.

3. The method of claim 2 , further comprising performing a principal component analysis (PCA) on the annotated images to reduce dimensionality and generating a trained model using a support vector machine method as a classifier.

4. 4. The method of claim 3, further comprising training the binary classification data with a convolutional neural network (CNN) to generate a trained model.

5. The method of claim 4, further comprising: deploying a trained model based on SVM and CNN to detect road roughness levels and classify the levels as good, average or bad.

Citation Information

Patent Citations

  • Highway traffic parameter real-time estimation method based on distributed optical fibers

    CN107591002A

  • Method and system for automated sequencing of vehicles in side-by-side drive-thru configurations via appearance-based classification

    JP2015210824A

  • System and method for monitoring strain in roadway optical cable

    US20180266902A1

  • Structure deterioration detection system, and structure deterioration detection method

    WO2020194494A1

  • Distributed intelligent traffic informatics using fiber sensing

    WO2020257167A1