Contrastive learning of utility pole representations from distributed acoustic sensing signals

Abstract

The experimental procedure includes a data collection procedure and a contrastive learning-based approach to establish the profile of the surveyed utility poles in the embedding space. The unique characteristics of the multiple utility poles are preserved in a low-dimensional feature vector. The similarity between pairs of samples collected on the same or different utility poles is reflected by the Euclidean distance between the embeddings of the utility poles. During data collection, the variations in the excitation signal, e.g., impact intensity on the DFOS / DAS optical sensor fiber / cable, impact location, ambiguity in impact time, ambiguity in data collection location, etc., are manually incorporated. The data collected in this way provides a model learning with complete information about the utility poles and is more robust with respect to uncontrollable factors during operation.

Description

[Technical field]

[0001] This disclosure relates generally to distributed optical fiber sensing (DFOS) systems, methods and structures. More particularly, the present invention relates to a method for contrastive learning of utility pole representations from distributed acoustic sensing (DAS) signals generated during operation of a DFOS / DAS system. [Background technology]

[0002] As can be readily appreciated by those skilled in the art, distributed optical fiber sensing systems and methods provide a variety of useful services, such as sensing a variety of physical parameters including temperature, vibration, strain, etc., thereby enabling a new era of infrastructure monitoring.

[0003] Distributed acoustic sensing techniques measure strain changes (stretch or compression) in the core of an optical fiber that is typically protected by an outer jacket / cladding material and attached / suspended on a utility pole by a fixture structure. Integrity assessment or structural health monitoring of such utility poles can be done based on the response signal of manual excitation (hammer or frequency sweep) of the pole. However, the DAS signal captured near the pole may contain not only information on the properties of the pole on which the optical sensing fiber is attached / suspended, but also information from the main excitation signal, vibration signals propagating along the optical sensing fiber from far away (e.g. transformers), or environmental noise (e.g. wind or traffic) all mixed together.

[0004] Given the usefulness of DAS technology, it is important to extract intrinsic information of utility poles and remove the effects of exogenous factors from a limited amount of data. However, it is difficult to ground-truth these factors during data collection, and it is generally prohibitive to conduct experiments that consider all possible factor combinations. Also, due to the large amount of DAS waveform data and the high variability of utility poles, there are no signal filters or known pre-processing procedures that can achieve this goal. Summary of the Invention

[0005] An advancement in the art is made according to aspects of the present disclosure directed to a method including a data collection procedure and a contrastive learning based approach for establishing a profile of each surveyed utility pole in an embedding space. The unique characteristics of the utility pole are preserved in a low dimensional feature vector. The similarity between pairs of samples collected on the same or different utility poles is reflected by the Euclidean distance between the embeddings of the utility poles.

[0006] According to an aspect of the present disclosure, the variation of the excitation signal, such as the impact intensity on the DFOS optical sensor fiber / cable, the impact location, the ambiguity of the impact time, and the ambiguity of the data collection location are manually introduced during data collection. The data collected in this way provides a learning model that has learned complete information about the utility pole and is more robust with respect to uncontrollable factors during operation.

[0007] According to a further aspect of the present disclosure, the inventors have developed a model training procedure that effectively extracts the intrinsic characteristics of utility poles (e.g., structural integrity, dimensions, structural diversity) and remote exogenous influences (e.g., excitation strength, weather conditions, road traffic) without knowing the ground truth of these factors. The only identifying label required is the ID of the tested utility pole, which is readily available. The model is trained adaptively, end-to-end, and advantageously is easy to implement on modern deep learning frameworks such as PyTorch. [Brief description of the drawings]

[0008] A more complete understanding of the present disclosure may be realized by reference to the accompanying drawings.

[0009] [Figure 1] 1 is a schematic diagram of an exemplary prior art distributed fiber optic sensing system.

[0010] [Diagram 2] FIG. 1 is a schematic diagram illustrating a collected DAS signal affected by several intrinsic and extrinsic factors, according to an embodiment of the present disclosure.

[0011] [Diagram 3] FIG. 1 is a schematic diagram illustrating a flowchart of a deep representation learning approach according to an aspect of the present disclosure.

[0012] [Figure 4] 1 is a plot illustrating an example utility pole profile map, according to an aspect of the present disclosure.

[0013] [Diagram 5] 1 is a plot illustrating an example utility pole profile map illustrating an example selective classification on a learned profile map, according to aspects of the present disclosure.

[0014] [Figure 6] FIG. 1 illustrates selective classification based on learned embeddings, including a confusion matrix from the original KNN classification, a classification by rule (10,0), and a classification by rule (2,1), according to an embodiment of the present disclosure.

[0015] [Figure 7] 1 is a plot illustrating an exemplary pole profile map showing the change in condition of seven "crust rot" poles from one path that were initially surrounded by poles of incipient corrosion, which after repair are considered more "normal" in accordance with an aspect of the present disclosure.

[0016] [Figure 8] 11 is a plot illustrating an example utility pole data collection scheme involving mechanical knocking with two different intensities (strong and weak) at three different heights parallel and perpendicular to the fiber optic sensor cable direction, according to an embodiment of the present disclosure.

[0017] [Figure 9] FIG. 1 illustrates an example model architecture of a neural network used for utility pole profiling, according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Exemplary embodiments are more fully described in the drawings and detailed description, however, embodiments in accordance with the present disclosure may be embodied in various forms and are not limited to the specific or exemplary embodiments set forth in the drawings and detailed description.

[0019] The following merely illustrates the principles of the present disclosure, and it will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the present disclosure and are included within its spirit and scope.

[0020] Furthermore, all examples and conditional terms described herein are intended to be for educational purposes only to aid the reader in understanding the concepts contributed by the inventors to further the principles and techniques of the present disclosure, and should not be construed as being limited to such specifically recited examples and conditions.

[0021] Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., elements developed that perform the same function, regardless of structure.

[0022] Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure.

[0023] Unless otherwise specified herein, the figures making up the drawings are not drawn to scale.

[0024] As some additional background, note that a distributed fiber optic sensing system interconnects an optoelectronic integrator to an optical fiber (or cable), transforming the fiber into an array of sensors distributed along the length of the fiber. In effect, the fiber becomes the sensor, and the interrogator generates / injects laser light energy into the fiber to sense / detect events along the fiber length.

[0025] As those skilled in the art will understand and appreciate, DFOS technology can be deployed to continuously monitor vehicle movement, human traffic, drilling activity, seismic activity, temperature, structural integrity, liquid and gas leaks, and many other conditions and activities. It is used worldwide to monitor power plants, communication networks, railroads, roads, bridges, borders, critical infrastructure, onshore and undersea power and pipelines, and downhole applications in oil, gas, and enhanced geothermal power generation. Advantageously, distributed fiber optic sensing is not constrained by line of sight or remote power access, and depending on the system configuration, can be deployed at continuous lengths of over 30 miles, with sensing / detection possible at all points along that length. Thus, the cost per sensing point over long distances is typically incomparable to competing technologies.

[0026] Fiber optic sensing measures changes in the "backscatter" of light that occurs in an optical sensing fiber when the sensing fiber encounters a vibration, strain, or temperature change event. As mentioned above, the sensing fiber acts as a sensor along its entire length, providing real-time information about the physical / environmental surroundings, and the integrity / security of the fiber. Additionally, distributed fiber optic sensing data pinpoints the exact location of events and conditions occurring at or near the sensing fiber.

[0027] A schematic diagram showing a generalized configuration and operation of a prior art distributed optical fiber sensing system is shown in Figure 1. With reference to Figure 1, one can observe an optical sensing fiber connected to an interrogator. As is known, a modern interrogator is a system that generates an input signal to the fiber and detects / analyzes the received signal after being reflected / scattered. The signal is analyzed and an output is generated that is indicative of the environmental conditions encountered along the length of the fiber. The signal thus received may result from reflections within the fiber such as Raman backscattering, Rayleigh backscattering, and Brillion backscattering. It may also be a forward signal that exploits the velocity difference of multiple modes. Without loss of generality, the following description assumes a reflected signal, but the same approach can be applied to a forward signal as well.

[0028] As will be appreciated, modern DFOS systems include an interrogator that periodically generates optical pulses (or any coded signal) and injects them into an optical fiber. The injected optical pulse signal is transmitted along the optical fiber.

[0029] At each location along the length of the fiber, a small portion of the signal is scattered / reflected back to the interrogator. The scattered / reflected signal carries information that the interrogator uses to detect, e.g., changes in power level indicative of mechanical vibrations.

[0030] The reflected signal is converted to the electrical domain and processed inside the interrogator. Based on the time of pulse injection and the time the signal is detected, the interrogator can determine from which location along the fiber the signal came, and thus sense activity at each location along the fiber.

[0031] Distributed Acoustic Sensing (DAS) / Distributed Vibration Sensing (DVS) systems detect vibrations and capture acoustic energy along a length of optical sensing fiber. Advantageously, existing, traffic-carrying optical fiber networks can be utilized to convert into distributed acoustic sensors and capture real-time data. Additionally, classification algorithms can be used to detect and locate events such as leaks, cable faults, intrusive activity, or other anomalous events involving both acoustic and / or vibration.

[0032] Currently, various DAS / DVS techniques are used, the most common being based on coherent optical time domain reflectometry (C-OTDR). C-OTDR utilizes Rayleigh backscattering and can detect acoustic frequency signals over long distances. The interrogator transmits coherent laser pulses along the length of an optical sensor fiber (cable). Due to scattering sites in the fiber, the fiber acts as a dispersive interferometer with a gauge length equal to the pulse length (e.g. 10 meters). Acoustic disturbances acting on the sensor fiber cause microscopic stretching or compression (microstrain) of the fiber, resulting in changes in the phase relationship and / or amplitude of the optical pulses traveling in the fiber.

[0033] Before the next laser pulse is sent, the previous pulse must have time to travel the entire length of the sensing fiber and its scattering / reflection back. Thus, the maximum pulse rate is determined by the length of the fiber. It is therefore possible to measure acoustic signals that vary in frequency up to the Nyquist frequency, which is usually half the pulse rate. Higher frequencies decay very quickly, so most of the frequencies relevant for event detection and classification are in the low 2kHz range.

[0034] As mentioned before, distributed acoustic sensing (DAS) technology measures the strain changes (stretch or compression) of an optical sensor fiber core, which is typically included as part of a larger cable assembly protected by an outer jacket and attached / suspended to a utility pole by a fixture structure. Integrity assessment or structural health monitoring of the utility pole can be done based on the response signal of manual excitation (hammer or frequency sweep) performed on the utility pole.

[0035] However, it should be noted that a DAS signal captured in close proximity to a utility pole may contain a mixture of information not only from the pole characteristics but also from the main excitation signal, vibration signals propagating along the sensor fiber cable from relatively far away points (e.g., transformers), or environmental noise (e.g., wind or traffic).

[0036] It is interesting to extract intrinsic information of utility poles and remove the influence of exogenous factors from a limited amount of data. However, it is difficult to know the ground truth of these factors during data collection, and it is generally prohibitive to conduct experiments covering all possible combinations of factors. Moreover, due to the large volume of DAS waveform data and the high variability that occurs on deployed utility poles, there are no hand-crafted signal filters or known pre-processing procedures that can achieve this goal. The performance of downstream algorithms depends heavily on the choice of representation.

[0037] With this background, it is noted that the method of the present invention includes both a data collection procedure and a contrastive learning-based approach to establish a profile of each surveyed utility pole in the embedding space. The unique characteristics of individual utility poles are preserved in low-dimensional feature vectors. The similarity between pairs of samples collected on the same or different utility poles is reflected by the Euclidean distance between the embeddings of the utility poles.

[0038] In our data collection procedure, we manually incorporate variations in the excitation signal, e.g., ambiguity in impact intensity on the optical sensor cable, impact location, impact time ambiguity, data collection location ambiguity, etc. This data design ensures that the trained model contains more complete information about the utility pole, resulting in data that is more robust to uncontrollable factors.

[0039] Our model training procedure effectively extracts the intrinsic characteristics of utility poles (e.g., structural integrity, dimensions, structural diversity) and remote exogenous influences (e.g., excitation strength, weather conditions, road traffic) without knowing the ground truth of these factors. The only identification label required is the ID of any inspected utility pole, which is readily available. The model is then trained adaptively and end-to-end. Advantageously, our approach can be easily implemented in modern deep learning frameworks such as PyTorch.

[0040] As will be shown and described, the inventive approach learns unique features about pole characteristics from DAS signals, which facilitate downstream tasks including classification of pole integrity based on human contractor labels, repair quality assessment after corroded poles are repaired / replaced, and detection of condition changes after extreme weather, earthquakes, or other events.

[0041] The inventive approach establishes a digital pole profile based on the frequency response to the impact force recorded by the DAS and derived by machine learning. The inventive approach advantageously obtains a complete information profile of each pole when multiple manual excitations are imposed at different locations on the pole and repeated multiple times.

[0042] As will be further explained, the inventive approach uses a deep metric learning approach that is trained with a supervised learning signal that makes the embeddings of pairs of samples collected from the same utility pole closer than those of pairs of samples collected from different utility poles. This choice reduces the size of the training data from O(NK) to O(N ∧ 2K ∧3), where K is the number of samples per utility pole and N is the number of utility poles. A variant of triplet loss is used to automatically mine hard pairs during the training phase and avoid learning plateaus. Advantageously, our approach preserves the neighborhood structure while mapping from the original high-dimensional data space to a low-dimensional embedding space.

[0043] Operationally, the learned utility pole embeddings are used for downstream security classification tasks. In particular, the learned neighborhood structure is used to perform selective classification, i.e., the model performs classification only for samples for which there is a sufficiently high confidence, while rejecting samples with low confidence. This is crucial for applications that need to meet high security and reliability criteria.

[0044] Misalignment between embedments of the same pole before and after a significant event (such as extreme weather, earthquakes, or repairs) suggests that the condition of the pole or its connection to its cable may have changed. The direction of change (worsening or improving) can be interpreted by looking at nearest neighbors. This feature provides useful information for preventive maintenance screening (e.g., suggesting which paths to prioritize for a more comprehensive inspection) and for objective assessment of repair quality.

[0045] FIG. 2 is a schematic diagram illustrating a collected DAS signal affected by several intrinsic and extrinsic factors, according to an embodiment of the present disclosure.

[0046] With reference to this diagram, it is noted that for purposes of the present invention, an individual utility pole is considered to be a system of utility poles in which an impact, such as an acoustic event or a direct mechanical impact, will cause vibrations in the affected utility pole. Such vibration impacts will result in a frequency response that may be detected by the DAS system.

[0047] The collected DAS signals are affected by several intrinsic and extrinsic factors. Intrinsic factors can include items such as pole integrity, pole dimensions, pole material, while extrinsic factors can include items such as pole location, excitation location, environmental factors such as wind, weather, or the presence or absence of transformers or other equipment (including electrical and / or other cables / wires suspended therefrom), as well as proximity to traffic or other vibration generating activity, and geological factors such as soil composition. From these frequency responses, a representation is needed to extract the intrinsic characteristics unique to each pole.

[0048] 3 is a schematic diagram showing a flow chart of the inventive deep representation learning approach according to an embodiment of the present disclosure. With reference to this figure, the overall operational flow of the inventive method according to an embodiment of the present disclosure can be observed.

[0049] In operation, the procedure of the present invention as shown in the flow chart begins with receiving at least three DAS signals, two of which originate from the same utility pole among a plurality of utility poles. The DAS signals are pre-processed and transformed from time domain to frequency domain. Using this converted / transformed DAS data, a deep representation learning process proceeds, performing a forward pass to determine / compute embeddings, followed by a distance loss determination, followed by backpropagation to update the neural network. This deep representation learning proceeds until sufficient convergence is achieved, resulting in a set of low-dimensional embedding vectors for downstream tasks.

[0050] 4 is a plot showing an exemplary utility pole profile map according to an embodiment of the present disclosure. This exemplary utility pole profile map is of 70 utility poles plotted on a map with our utility pole profiles. For this map, we used a training set of 44 utility poles and a test set of 26 utility poles. Given a random data point, our AI model can determine which utility pole the data point belongs to with 93% accuracy.

[0051] An application example of selective evaluation of utility pole maintenance based on reliability. The utility pole maintenance label can be predicted based on the learned utility pole embedding. The system of the present invention can be used as a pre-screening tool for utility pole maintenance. Due to the limited training size, the training classifier is not perfect, but the error cost and accuracy benefits are different.

[0052] A good pole, identified as "good", is a pole that does not require / subject to manual inspection, requires high reliability, and ultimately contributes significantly to the costs saved by the model of the present invention. A bad pole, identified as "bad", is a pole that is confirmed by manual inspection. A bad pole, identified as "good", is a missed alarm, may be a potential hazard, and needs to be minimized. A good pole, identified as "bad", is a false alarm, which will be subject to manual inspection and will cause costs that are not saved. In this application, a false alarm is not as critical as a missed alarm.

[0053] Based on the neighborhood structure learned in the profile map, we filter out uncertain samples using a (k,k') nearest neighbor classification rule and a pruning option. The model considers k nearest neighbors and prunes a neighbor if less than k' of these are from the same class.

[0054] 5 is a plot showing an example selective classification on a learned profile map according to an embodiment of the present disclosure, showing an example of the identity of utility poles. As depicted in the plot, black circles are classified as bad utility poles and black squares are classified as good utility poles. Conversely, crosses depicted in the plot are rejected by the model.

[0055] 6 is a schematic diagram illustrating selective classification based on learned embeddings according to an embodiment of the present disclosure. As shown in the figure, selective classification based on learned embeddings. The confusion matrix from the original KNN classification is shown at the top, while classification by rule (10,9) is shown on the left and classification by rule (2,1) is shown on the right.

[0056] Application example (change in condition of utility pole after repair)

[0057] To evaluate the method of the present invention, hammer knock data was collected from seven "bad" utility poles (crust rot) before and after repair, and the data was visualized along with data collected from other utility poles (normal, early corrosion). In the learned latent space, new samples are found that "shift" from their original positions. More specifically, such samples more closely resemble "normal" utility poles. Advantageously, profile maps according to aspects of the present disclosure can be used to evaluate whether utility pole repairs are effective.

[0058] FIG. 7 is a plot showing the change in condition of seven "shell rot external" poles after repair, according to an embodiment of the disclosure. The seven poles are initially surrounded by "early rot" poles. After repair, they "move across the decision boundary" and more closely resemble "normal" poles. The decision boundary between the "normal" poles and the "early rot, shell rot" poles is shown in the exemplary plot by different shading of the circles.

[0059] FIG. 8 is a schematic diagram illustrating an exemplary data collection scheme performed on a typical collection of utility poles that includes knocking with two different intensities (strong and weak) at three different heights parallel and perpendicular to the cable direction.

[0060] We will now describe the steps involved in the method of the present invention according to the embodiment of the present disclosure outlined in the flow diagram of FIG.

[0061] Step 1 of the method involves data collection of DAS data obtained from hammer knock-on (normal, incipient corrosion, crust rot) utility poles and 5-11 cable points around each utility pole. Exemplary knocking locations are shown in Figure 7. A metronome timer is used to calibrate the timing of each knock so that adjacent knocks do not overlap. In the peak detection algorithm, each sample is extracted from a raw waveform containing approximately 6.9 seconds of DAS data at 8 kHz (approximately 0.4 seconds before the knock and approximately 6.5 seconds after the knock). As shown in the flow diagram, the deep representation methodology of the present invention includes three separate DAS signals, two of which must originate from a common excited utility pole.

[0062] Step 2 of the method is a pre-processing step that converts the waveform data into a time-frequency domain representation by applying a Short-Time Fourier Transform (STFT) with a Hann window of size 2048. The resulting spectrogram is truncated to a maximum of 500 Hz.

[0063] Step 3 of the method is the data augmentation step, where three of the five cable locations, two replicates of four knocks at the same location, and four of the six knock locations on the pole are randomly sampled during each epoch of training to prevent overfitting or falling into local optima.

[0064] Step 4 of the method is the control training step. Assume that K samples are collected from each of the P poles. Two samples from the same pole and one sample from another pole form a triplet (with an anchor, positive, and negative). The total number of triplets is PK ∧ 2(P-1)×(K-1), most of which are redundant and rarely used after the first few epochs. We apply the following “batch-hard” triplet loss to facilitate training:

number

[0065] After deep metric learning, the time-frequency spectral data is projected into a low-dimensional embedding space, which can preserve the unique characteristics of each utility pole, i.e., samples collected from the same utility pole will be close to each other.

[0066] Step 5 of the method of the present invention may include secondary tasks. Such secondary machine learning tasks may be performed based on needs and include training a KNN classifier based on the learned embeddings to assess the integrity of the poles, storing the embeddings as low-dimensional features and then visualizing them on a profile map, removing uncertain samples based on the neighborhood structure of the embedding space, and identifying or tracking poles that exhibit a change in condition based on the similarity of the query pole to historical data collected from the same pole.

[0067] FIG. 9 is a schematic diagram illustrating an example model architecture of a neural network used in accordance with aspects of the present disclosure.

[0068] At this point, the present disclosure has been presented using several specific examples, but those skilled in the art will recognize that the present teachings are not so limited. Accordingly, the present disclosure should be limited only by the scope of the claims appended hereto.

Claims

1. 1. A method for operating a distributed fiber optic sensing (DFOS) / distributed acoustic sensing (DAS) system including an optical sensing fiber suspended in the air by a plurality of utility poles and in optical communication with a DFOS interrogator / analyzer, the method comprising: operating said DFOS / DAS system while manually exciting one or more utility poles to obtain a frequency response of the excited utility poles; Contrastively training a convolutional neural network (CNN) using the obtained frequency responses; and classifying the plurality of utility poles using the contrastively trained CNN; and generating a profile map of the excited poles indicative of the classified poles; The method wherein the manual excitation of each of the one or more utility poles includes mechanical shocks at three different positions in the height direction of the utility pole, the mechanical shocks at each position having two different intensities, one of the two different intensities being applied as a mechanical shock parallel to an orientation direction in which the optical sensing fiber suspended in the air is oriented, and the other of the two different intensities being applied as a mechanical shock perpendicular to the orientation direction.

2. 2. The method of claim 1, wherein the frequency response of the excited utility pole is transformed into the time-frequency domain by applying a Short-Time Fourier Transform (STFT) with a Hann window of size 2048 such that the resulting spectrogram is truncated to 500 Hz.

3. The method of claim 1 , wherein four of six manual excitations performed on an individual pole are randomly sampled for subsequent training and / or classification.

4. The method of claim 3 , wherein at least two poles of the plurality of poles are manually mechanically excited for the training and subsequent classification.

5. The method of claim 1 , wherein the mechanical shocks are applied with varying time intervals between shocks.

6. 6. The method of claim 5, wherein the classification of the mechanically impacted pole of the plurality of poles indicates a pole condition selected from the group consisting of normal, incipient corrosion, framework corrosion, and repair.

7. K mechanical shocks (samples) are performed from each of the P poles, and two samples from the same pole and one sample from another pole form triplets, so that the total number of triplets is P ∧ 2(P-1)×(K-1), and the triplet loss is [0010] The method according to claim 4, which is carried out according to

Images