Dynamic Acoustic Signature System with Sensor Fusion for Illegal Logging in Rain Forests

Abstract

Example implementations described herein may be directed to detecting anthropogenic disturbances from sensor data streaming from one or more sensors in a network of sensors, processing the detected anthropogenic disturbances to determine a probability of a chainsaw event (e.g., a light chainsaw event or a dense chainsaw event) for an area associated with the one or more sensors and an estimated lead time of the chainsaw event, and determining, for neighboring sensors to one or more sensors in the network of sensors, a probability of a state change to an anthropogenic disturbance or a chainsaw event in other areas associated with the neighboring sensors.

Description

[Technical field]

[0001] The present disclosure is directed generally to Internet of Things (IoT) systems, and more particularly to IoT systems that facilitate dynamic acoustic signature recognition. [Background technology]

[0002] Illegal logging is the cutting of forests in violation of natural regulations. The environmental impacts of illegal logging include deforestation, biodiversity loss and greenhouse gas emissions.

[0003] The sheer size of the rainforest wilderness increases the difficulty of locating illegal loggers. Even with several acoustic sensors collecting data, environmental noise and intermingling animal sounds make the rainforest more difficult to analyze.

[0004] New IoT technologies, advanced machine learning systems and algorithms are being developed and applied to combat illegal logging, but the implementation of conventional technologies faces significant limitations.

[0005] One limitation is the insufficiency of acoustic labels. Prior art systems rely on accurate labels and sensor data that are difficult to derive for several reasons. First, manual labeling processes can produce relatively accurate labels, but such processes consume significant human resources. Second, historical records may not cover all scenarios. Third, new anomalous events cannot be identified or prevented when there is insufficient data.

[0006] Another problem of the prior art is the lack of deterrence prediction. Prior art approaches detect illegal logging only when it is taking place. By the time the guards discover the logging site, the illegal loggers have already left. In other words, there is no preventative action. The various predictive labels lack temporal and causal relationships. The mechanism for relating the labels to the lead times is pre-determined / pre-computed and therefore cannot be updated with new information.

[0007] Furthermore, the reliability of prior art models is highly dependent on the completeness of data and the operational time of sensors. Missing data or sensor failures will always degrade the model's performance or prevent it from running. Existing models lack fault tolerance and cannot identify root causes or false negative records after deployment. Due to environmental constraints, sensor-based data may show environmental noise differently, which interferes with the deployed pre-learned patterns. Summary of the Invention [Means for solving the problem]

[0008] To address the above-mentioned problems, implementations described herein are directed to a media signature system to label acoustic signals or images from fused sensors across the Guardian to identify leading indicators of illegal logging and predict logger traversal patterns to stop illegal logging before it occurs.

[0009] The implementation addresses label availability by using a dynamic fingerprinting system. By extracting maximum information from a single channel acoustic file or image in real time, the system creates a label using only the acoustic signal. Sensor availability is addressed by using synthetic data that combines anomalous and background signals with adjusted power levels and modified harmonics.

[0010] Additionally, implementations described herein identify signal fingerprints within multi-source complex and continuous background noise. Media fingerprinting identifies temporal patterns of audio signals or images from a single channel based on their perceptual characteristics. Implementations described herein incorporate event correlation between anomalies and their sequences, thereby identifying leading indicators of specific anomalous audio signals or images. As a result, implementations can generate actionable alerts by using leading indicators for potential anomalous signals.

[0011] Finally, this implementation predicts anomaly footprints that consider sensor data and / or known characteristics of the entire environment. Media footprinting identifies traversal patterns that consider environmental and geographic information across all channels. Media footprinting integrates information across sensors to create a fault-tolerant model to address rainforest environments through sensor fusion techniques. Sensor fusion is the combination and aggregation of sensor data or data derived from sensor data such that the resulting information is better than using the data alone. Sensor fusion improves analysis algorithms with fault tolerance because it uses environmental context information and human and sensor inputs as a priori knowledge and current state, and does not have to rely on a specific sensor or group of sensors to determine analysis results. By building a self-feedback, error-learning system with limited human intervention, this technique also improves labeling results.

[0012] Thus, implementations can address limitations of the prior art and dynamically predict acoustic signatures using sensor fusion.

[0013] Aspects of the present disclosure include methods that may include identifying and labeling patterns based on characteristics of sensor data streaming from one or more sensors in a network of sensors, processing the identified and labeled patterns to determine a probability of a chainsaw event and an estimated lead time of the chainsaw event for an area associated with the one or more sensors, and determining, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a human disturbance or state change to a chainsaw event in other areas associated with the neighboring sensors.

[0014] Aspects of the present disclosure include a computer program storing instructions that may include identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors, processing the identified and labeled patterns to determine a probability of a chainsaw event and an estimated lead time of the chainsaw event for an area associated with the one or more sensors, and determining, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a human disturbance or state change to a chainsaw event for other areas associated with the neighboring sensors. The instructions may be stored in a non-transitory computer readable medium and configured to be executed by one or more processors.

[0015] Aspects of the present disclosure include a system that may include means for identifying and labeling patterns based on characteristics of sensor data streaming from one or more sensors in a network of sensors, means for processing the identified and labeled patterns to determine a probability of a chainsaw event and an estimated lead time of a chainsaw event for an area associated with the one or more sensors, and means for determining, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a human disturbance or state change to a chainsaw event in other areas associated with the neighboring sensors.

[0016] Aspects of the present disclosure include an apparatus including a processor configured to identify and label patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors, process the identified and labeled patterns to determine a probability of a chainsaw event and an estimated lead time of the chainsaw event for an area associated with the one or more sensors, and determine, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a human-induced disturbance or state change to a chainsaw event in other areas associated with the neighboring sensors.

[0017] Aspects of the disclosure may include a method that may include detecting a first event from sensor data streamed from one or more sensors in a network of sensors, processing the first event to determine a probability of a second event for an area associated with the one or more sensors and an estimated lead time of the second event, and determining, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a state change to the first event or the second event in other areas associated with the neighboring sensors.

[0018] Aspects of the present disclosure may include a computer program having instructions that may include detecting a first event from sensor data streamed from one or more sensors in a network of sensors, processing the first event to determine a probability of a second event for an area associated with the one or more sensors and an estimated lead time of the second event, and determining for neighboring sensors to the one or more sensors in the network of sensors a probability of a state change to the first event or the second event for other areas associated with the neighboring sensors. The instructions may be stored in a non-transitory computer readable medium and executed by one or more processors.

[0019] Aspects of the disclosure may include a system that may include means for detecting a first event from sensor data streaming from one or more sensors in a network of sensors, means for processing the first event to determine a probability of a second event and an estimated lead time of the second event for an area associated with the one or more sensors, and means for determining, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a state change to the first event or the second event in other areas associated with the neighboring sensors.

[0020] Aspects of the disclosure may include an apparatus that may include a processor configured to detect a first event from sensor data streamed from one or more sensors in a network of sensors, process the first event to determine a probability of a second event for an area associated with the one or more sensors and an estimated lead time of the second event, and determine, for neighboring sensors to the one or more sensors in the network of sensors, a probability of a state change to the first event or the second event in other areas associated with the neighboring sensors.

[0021] Aspects of the present disclosure may include a method that may include identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors, processing the identified and labeled patterns to estimate probabilities and lead times for changes in areas associated with the one or more sensors from a current sensor stage to another sensor stage, including a sequential error learning process configured to improve accuracy, and determining, for neighboring sensors to one or more sensors in the network of sensors, probabilities of sensor stage changes in other areas associated with the neighboring sensors.

[0022] Aspects of the present disclosure may include a system that may include means for identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors; means for processing the identified and labeled patterns to estimate probabilities and lead times for changes in areas associated with the one or more sensors from a current sensor stage to another sensor stage, including a sequential error learning process configured to improve accuracy; and means for determining, for neighboring sensors to one or more sensors in the network of sensors, probabilities of sensor stage changes in other areas associated with the neighboring sensors.

[0023] Aspects of the present disclosure may include a computer program that may include instructions that may include identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors, processing the identified and labeled patterns to estimate probabilities and lead times for a change in a region associated with one or more sensors from a current sensor stage to another sensor stage, including a sequential error learning process configured to improve accuracy, and determining, for neighboring sensors to one or more sensors in the network of sensors, probabilities of a change in sensor stage of other regions associated with the neighboring sensors. The instructions may be stored on a non-transitory computer readable medium and executed by one or more processors.

[0024] Aspects of the present disclosure may include an apparatus that may include a processor configured to identify and label patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors, process the identified and labeled patterns to estimate probabilities and lead times for changes in areas associated with one or more sensors from a current sensor stage to another sensor stage, including a sequential error learning process configured to improve accuracy, and determine, for neighboring sensors to one or more sensors in the network of sensors, probabilities of sensor stage changes in other areas associated with the neighboring sensors. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 illustrates an example of a general system for implementing information fusion, according to one implementation.

[0026] [Diagram 2] FIG. 2 illustrates an example of structured data and composite data, according to one implementation.

[0027] [Diagram 3]FIG. 3 shows an example of information structuring and processing according to one implementation.

[0028] [Figure 4] FIG. 4 illustrates an example of combining data with real-time acoustic signals and external events according to one implementation.

[0029] [Diagram 5] FIG. 5 illustrates an example of eligibility criteria, according to one implementation.

[0030] [Figure 6] FIG. 6 illustrates an example of information extraction from a single channel, according to one implementation.

[0031] [Figure 7] FIG. 7 illustrates an example of clustering acoustic signals from a single channel according to one implementation.

[0032] [Figure 8] FIG. 8 illustrates dynamic fingerprint analysis according to one implementation example.

[0033] [Figure 9] FIG. 9 shows an example of deriving a frequency spectrum of cluster centers according to one implementation.

[0034] [Figure 10] FIG. 10 illustrates the workflow of temporal fingerprint aggregation analysis according to one implementation example.

[0035] [Figure 11] FIG. 11 shows an example of the results using UTC time and GMT+7 local time zone according to one implementation.

[0036] [Figure 12] FIG. 12 illustrates an example of a predictive cluster according to one implementation.

[0037] [Figure 13]FIG. 13 illustrates an example of incorporating labels and lead times into a probability process, according to one implementation.

[0038] [Figure 14] FIG. 14 shows an example of using a discrete-time Markov chain as a stochastic process, according to one implementation.

[0039] [Figure 15] FIG. 15 illustrates an example of sequential error learning, according to one implementation.

[0040] [Figure 16] FIG. 16 illustrates an example of a sensor fusion construct and footprint traversal according to one implementation.

[0041] [Figure 17] FIG. 17 illustrates an example of mapping fingerprints of illegal logging, according to one implementation.

[0042] [Figure 18] FIG. 18 illustrates an example of traversing an illegal logging footprint according to one implementation.

[0043] [Figure 19] FIG. 19 illustrates a system including multiple systems with connected sensors and management devices, according to one implementation.

[0044] [Figure 20] FIG. 20 illustrates an example computing environment having an example computer device suitable for use with some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] The following detailed description shows the details of the drawings and implementation examples of the present application. Reference numbers and descriptions of redundant elements between the drawings are omitted for clarity. The terms used throughout the description are shown by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic implementation or a semi-automatic implementation that includes user or administrator control over certain aspects of the implementation, depending on the desired implementation of the person skilled in the art practicing the implementation of the present application. The selection may be made by a user through a user interface or other input means, or may be implemented by a desired algorithm. The implementation examples described herein may be utilized alone or in combination, and the functionality of the implementation examples may be implemented by any means depending on the desired implementation.

[0046] 1 illustrates an example of a general system for implementing information fusion, according to one implementation. The system may include the following aspects:

[0047] Structured and Synthetic Data 100 is a component that integrates information, evaluates qualifications, and stores filtered data in a central database. Further details of this component are described with respect to Figs. 2-5. Single Channel Extraction 200 is a component that clusters acoustic signals from a single channel. Further details of this component are given with respect to Figs. 6-7. Dynamic Fingerprint Analysis 300 is a component that identifies temporal patterns in acoustic signals or images and creates pseudo labels. Further details of this component are described with respect to Figs. 8-12. Incorporating Labels and Lead Times into a Stochastic Process 400 is a component that builds a stochastic process to refine labels / lead times and learn patterns. Further details of this component are described with respect to Figs. 13-14. Sequential Error Learning 500 is a component that improves labeling results by creating a self-feedback loop with human reviewed samples. Further details of this component are described with respect to Fig. 15. The Sensor Fusion Construct and Footprint Traverse 600 is a component that predicts anomalous footprints by considering sensor data and / or known characteristics of the entire environment. Further details of this component are provided with respect to Figures 16-18.

[0048] The four functions of information fusion are shown in Figure 1, including comprehension (synthesizing and interpreting information), perception (identifying and detecting information), projection (predicting and simulating information), and resolution (making decisions and plans based on the information).

[0049] Figure 2 shows an example of structured and synthetic data 100 according to one implementation. The first step is to integrate 106 the information, evaluate the qualification 107 and store the filtered data in a central database. The workflow is shown in Figure 2. There are four potential data sources including real-time acoustic signals 101, subject matter expert (SME) descriptions 102, external events 103 and synthetic data 105.

[0050] As shown in Figure 3, it structures and processes information from a variety of inputs.

[0051] In contrast to the prior art, this component facilitates several novel aspects, including utilizing synthetic data to enhance the training dataset and assessing the quality of an audio file or image based on a combination of techniques.

[0052] The real-time image or audio files 101 include, but are not limited to, audio files 110, satellite images 111, camera images 112, and camera videos 113. Geographic and temporal metadata 114 are extracted and integrated from all data sources. Acoustic signals are collected using equipment placed within the rainforest and transferred over a network to a computational cloud. The acoustic signals record sounds from animals, species, humans, etc. within the rainforest, which include a spectrum of frequencies from a few Hertz (Hz) to thousands of Hz. In order to integrate this information, the audio files 110 need to be sliced ​​and stored in a specified format and frequency. Additionally, depending on the desired implementation, the satellite images 111, camera images 112, and camera videos 113 are alternatives or companions to the audio files 110. The camera videos 113 include both audio and video information, which gives more data dimensions. Using alternative data inputs may facilitate person and fingerprint detection more directly.

[0053] The SME description 102 is a response from a ranger in the Guardian of the Rainforest. When the detection model sends out an alert about illegal logging, a ranger can go and verify the illegal logging activity. After the ranger's verification on-site, the ranger can respond by describing his findings and with evidence. Natural Language Processing (NLP) 120 is applied to synthesize this information and create anomaly labels 121 with corresponding timestamps.

[0054] External events 103 are labeled acoustic files in a public dataset or a specific acoustic database. Labels might include, for example, "chainsaw," "barking dog," and "vehicle." Usage is illustrated in synthetic data.

[0055] In an example implementation, synthetic data 105 is generated. The purpose of using the synthetic data 105 is to augment the training dataset, particularly when the training dataset lacks examples of the target event or the data is skewed, the synthetic data 105 can be used to balance the data. The synthetic data is constructed 104 using real-time acoustic signals and external events.

[0056] FIG. 4 shows an example of synthesizing data with real-time acoustic signals and external events 103 according to one implementation. First, each of the acoustic examples is viewed separately. The first example type includes a noise signal 141 that serves as background noise. The second example type is for a target event 140 from a real-time anomalous file, and extracts signal details. The process then uses the extracted information to replicate the presence of the anomalous signal in the desired environment by modifying the signal power level 145, frequency 146, and harmonics 148. The process then modifies the signal power level relative to the background noise 142 to ensure that the real-time acoustic signal can be mimicked. The process then combines the anomalous signal 150 and the noise signal by superposition 151 and uses a generative adversarial network (GAN) 152 to verify 153 that the synthetic data is indistinguishable from the real-time data. Once verified, the synthetic data 105 can be kept.

[0057] FIG. 5 shows an example of eligibility criteria 107 according to one implementation example. After building the centralized data, audio and images need to be evaluated and filtered. The evaluation process is necessary to achieve higher accuracy for future stages. FIG. 5 shows a combined approach to select eligible components. For audio files, audio sizing 170 and sampling rate 171 are evaluated, while for images and videos, image sizing 172, image resolution 173, and image red-green-blue (RGB) and grayscale 174 are detected. Based on the audio and image detection, outliers are appropriately removed at 175.

[0058] Fig. 6 shows an example of information extraction from a single channel according to an implementation example. The second stage is to cluster the acoustic signals from a single channel. The workflow of this stage is shown in Fig. 6. Fig. 7 shows an example of clustering the acoustic signals from a single channel according to an implementation example.

[0059] First, at 201, the acoustic data is transformed from the time domain to the frequency domain. The frequency domain refers to an analytical space in which mathematical functions or signals are conveyed in terms of frequency rather than time. For example, a time domain graph may show changes over time, whereas a frequency domain graph shows how much of a signal is within each given frequency band.

[0060] In Figure 7, we operate on a Fast Fourier Transform (FTT), but there are alternatives such as Mel-Frequency Cepstral Coefficients (MFCC), Short-Time Fourier Transform (STFT), etc. that can be used according to the implementation example. Instead of using the bandwidth from the FFT result, a Kernel Density Estimation (KDE) is applied to smooth the amplitude results that come from the frequency domain result.

[0061] After deriving the frequency domain features, dimensionality reduction 202 can be applied to transform the data from a high-dimensional space to a low-dimensional space. Sound files in a rainforest typically contain a wide range of sound frequencies, from a few Hz (low frequencies, e.g., chainsaws, human speech) to thousands of Hz (high frequencies, e.g., birdsong, insects chirping). The low-dimensional representation ideally preserves some meaningful properties of the original data, close to its inherent dimensionality.

[0062] In Figure 7, Principal Component Analysis (PCA) is performed to project the original data into fewer dimensions. Depending on the desired implementation, other alternatives such as Non-negative Matrix Factorization (NMF), Linear Discriminant Analysis (LDA) can also be used.

[0063] Once the features for clustering are defined, we perform hyperparameter tuning 203 and clustering technique 204 to create cluster numbers. Clustering is the task of dividing a population or data points into groups such that data points in the same group are more similar to other data points in the same group than data points in other groups. In other words, the goal is to separate groups with similar characteristics and assign them into clusters. With input as amplitudes of frequency bands, clustering is to group the amplitudes to find similar frequency spectra and distributions.

[0064] In Figure 7, a Gaussian Mixture Model (GMM) is used. GMM is a probabilistic model that assumes that all data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. With each Gaussian distribution representing a cluster, the clustering process is to identify the probability of data belonging to each distribution. Thus, GMM provides soft clustering, while other hard clustering techniques such as K-means, hierarchical and density-based spatial clustering can also be applied depending on the desired implementation.

[0065] The result of this step is one cluster number per data point.

[0066] Figure 8 shows a dynamic fingerprint analysis 300 according to one implementation example. The third step is to identify the temporal patterns of an audio signal or image based on its perceptual characteristics from a single channel. In other words, knowing the cluster number is not enough to label an audio signal. Fingerprint analysis from a single channel requires consideration of frequency, temporal and association aspects.

[0067] Figure 8 shows the workflow of this stage. In this stage, frequency band analysis 301, temporal fingerprint aggregation analysis 302 and event association analysis 303 are comprehensively considered to generate pseudo labels 304 for the acoustic signals.

[0068] Compared to prior art implementations, this stage includes several novel aspects, for example, it identifies signal fingerprints within multi-source, complex and continuous background noise, and further identifies leading indicators for anomalous events, and further analyzes event associations between anomalies and their sequences through frequency band distribution and image classification.

[0069] Frequency band analysis 301 is used to analyze the frequency spectrum of the cluster centers with their underlying speech frequencies. A frequency band is an interval in the frequency domain bounded by low and high frequencies.

[0070] With the input from the previous stage, a reverse step can be used to derive the frequency spectrum of the cluster centers compared to the information extraction. The cluster centers, as a form of features, can be reverted by the dimensionality reduction model and reconstructed into the original frequency domain.

[0071] Figure 9 shows an example of deriving a frequency spectrum of cluster centers according to one implementation. There are four cluster centers that have been transformed back to the frequency domain. The X-axis is frequency from 0 Hz to 6,000 Hz. The Y-axis is the probability density considering the frequency spectrum as a distribution. Low frequency sounds (human conversation and chainsaw sounds) and high frequency sounds (birds chirping and insects trilling) can be recognized and labels can be established for audio components.

[0072] 10 shows the workflow of temporal fingerprint aggregation analysis 302 according to one implementation example. Temporal fingerprint aggregation analysis is utilized at this stage. To identify temporal patterns, clusters are associated with timestamps 320 and aggregated 322 with different thresholds that are optimized using hyper-parameter tuning 321.

[0073] To illustrate how this can be used in practice, Figure 11 shows an example of results using UTC time and GMT+7 local time zone according to one implementation. The X-axis represents time, while the Y-axis represents cluster counts. A temporal pattern can be inferred: cluster center 1 appears during the day, before or after alternating cluster centers 2 and 3, while cluster center 4 dominates during the night.

[0074] Another implementation of this stage is the event association analysis. This novel step not only considers the event sequence from pre-event to event to post-event, but also uses frequency and time information to identify event attributes and classes.

[0075] In the rainforest use case, the life cycle of an illegal harvest can be described as original ecosystem, human disturbance, illegal harvest and disturbed ecosystem.

[0076] With the SME description as input, we can associate anomalous events with each cluster. The overlap between anomalous events and predicted clusters facilitates understanding the relationship between previous and subsequent clusters.

[0077] FIG. 12 shows an example of predicted clusters according to one implementation. The predicted clusters are marked with various shapes including circles, squares, diamonds, and triangles. Considering the 121 anomaly labels in FIG. 3, the shapes are filled with black if there are any anomalies extracted from the SME description in the signal period. In this example, the black dots are almost absent in cluster 1 (triangle) and cluster 4 (diamond), lightly present in cluster 3 (circle), and densely present in cluster 2 (square). Thus, cluster 2 represents the voice with the most chainsaw sounds identified by the SME, while cluster 3 represents the voice with fewer chainsaw sounds identified by the SME.

[0078] Furthermore, the information obtained from the frequency band analysis and the temporal fingerprint aggregation analysis can be combined to derive the following relationships along with the corresponding labels for each cluster: Cluster 1: Cluster of anthropogenic disturbances Cluster 2: Dense chainsaw cluster Cluster 3: Light chainsaw cluster Cluster 4: Quiet Time Cluster

[0079] As a result, pseudo labels are generated after this new stage. The definition of each cluster is as follows: The Anthropogenic Disturbance cluster identifies sounds with human activity occurring before and after logging activities. This cluster includes human activities such as searching the rainforest for logging candidates or hauling logs out of the rainforest. This cluster may consist of sounds such as human conversations, motorcycles, cars, barking dogs, etc. Due to anthropogenic disturbance, animal sounds in such sounds are rare compared to the quiet time cluster. The Dense Chainsaw cluster identifies audio with intense chainsaw sounds. This cluster occurs during logging activities with a clear and loud chainsaw sound taking up most of the length of the audio. This cluster represents actual logging times when the chainsaw sound dominates the audio while animal sounds are completely occluded. This cluster usually occurs during the day when loggers have a clear view of the rainforest. The light chainsaw cluster identifies sounds that have fewer chainsaw sounds. This cluster occurs during logging activities, but the chainsaw sounds occupy only a partial sound length. During logging activities, loggers may take breaks or change their tools. Such interruptions explain why fewer chainsaw sounds are detected. The light chainsaw label is usually clustered during the day when loggers have clear vision of the rainforest. The Quiet Time cluster identifies sounds that have natural sounds in the rainforest, including animal sounds, insect sounds, rain / wind sounds, etc. This cluster represents pure environmental sounds without human intervention. This cluster can occur during the day or night.

[0080] Figure 13 shows an example of incorporating 400 labels and lead times into a stochastic process according to one implementation. The fourth step is to build a stochastic process to refine the labels / lead times and learn patterns. Lead time is defined as the amount of time that passes from one label to the next. Due to the temporal nature of the labels, lead times can be calculated between different labels or the same label.

[0081] Figure 13 shows an example of a workflow for this stage. With pseudo labels as input, the labels can be refined at 401 and the lead times can be calculated at 404. Two neural network models can be generated (e.g., neural network labeling 403 and sequential deep learning 406) and the accuracy can be improved by hyper-parameter tuning 402 and 405. After learning the patterns and obtaining the prediction results, the predicted labels and lead times can be combined at 407 to create a stochastic process that can be configured to output the labels or lead times 408 as appropriate.

[0082] Compared to the prior art, this step is novel in order to build a multi-output process to incorporate both labels and lead times, and to improve the robustness and resilience of the model through a neural network model.

[0083] A Markov chain is a probabilistic model that undergoes transitions from one state to another according to certain probability rules. Figure 14 shows an example of using a discrete-time Markov chain as a stochastic process, according to one implementation. In a discrete-time Markov chain, the chain jumps from the current state to the next state after one unit of time.

[0084] In the rainforest use case, assume there are three clusters (A / B / C), where A is the quiet cluster, B is the anthropogenic disturbance cluster, and C is the chainsaw cluster.

[0085] For a single asset, we calculate the probability of cluster changes probabilistically from history. For example, with three defined clusters, the probability from t1 to t2 has the following nine combinations: · PAA: self-transition and probability of staying in cluster A. ·PAB: Possibility of state change from Cluster A to Cluster B. · PAC: Possibility of state change from Cluster A to Cluster C. · PBB: self-transition and probability of staying in cluster B. · PBA: Probability of state change from Cluster B to Cluster A. · PBC: Possibility of state change from Cluster B to Cluster C. · PCC: self-transition and probability of staying in cluster C. ·PCA: Probability of state change from cluster C to cluster A. · PCB: Possibility of state change from Cluster C to Cluster B.

[0086] For a single asset, the lead time of cluster change is calculated probabilistically from history. The same framework of Figure 14 can be used again while replacing the probability with lead time. For example, if there are records of t1, t2, t3 with corresponding states as cluster A, cluster A, cluster B, the lead time can be calculated and given as LAA (i.e. lead time from cluster A to cluster A) and LAB (i.e. lead time from cluster A to cluster B).

[0087] Thus, a continuous-time Markov chain has two components: first, there should be a discrete-time Markov chain giving the transition probabilities Pij, and second, there should be a reservation time parameter Lij for each state that controls the amount of time spent in each state.

[0088] A stochastic process can be created using the transition probabilities Pij and hold time parameters Lij to generate the predicted labels and lead times.

[0089] FIG. 15 shows an example of sequential error learning 500 according to one implementation. The fifth stage is to improve the labeling results by creating a self-feedback loop with human-reviewed samples. The workflow of this stage is shown in FIG. 15. As described herein, the sequential error learning process configured to improve accuracy may include a self-feedback loop configured to learn the error inherited from the stochastic process and improve the pseudo-labels between training phases.

[0090] At 501, obtain stratified samples based on the percentage of labels. At 502 and 503, review the samples (e.g., by a human or other external agent) and verify the predicted labels. At 504, if the error rate is below a threshold (yes), continue to the next stage; otherwise, if the error rate is equal to or above the threshold (no), train a multi-class classification model to learn the errors at 505. Once the classification model has learned about the errors, replace the pseudo labels in stage 3 with the predicted labels of the current stage at 506. The threshold can be set at any level depending on the desired implementation.

[0091] Due to the self-feedback loop for learning errors, this stage consistently improves the label results until the human review releases the label results. Thus, this stage can capture the errors inherited from the stochastic process with human intervention and correct and learn the errors sequentially. In addition, during the training phase, this stage can gradually improve the accuracy of the pseudo-labels and the model.

[0092] Compared with the prior art, this stage is novel because it improves the labeling results by building a self-feedback error learning system together with human-reviewed samples. As a result, this stage results in a sequential error learning system. If the error rate is satisfactory, this stage releases the label to the next stage.

[0093] 16 illustrates an example of a sensor fusion construct and footprint traversal 600, according to one implementation. The sixth step is to predict anomalous footprints that take into account the sensor data and / or known characteristics of the entire environment.

[0094] Sensor fusion 601 is the combination and aggregation of sensor data or data derived from sensor data such that the resulting information is better than using the data alone. Sensor fusion 601 provides a system level state to impute a sensor cohort of failed sensors or failed sensor information with prior temporal information.

[0095] Sensor Fusion 601 improves analysis algorithms with fault tolerance because it uses environmental contextual information and human and sensor inputs as a priori knowledge and current state, and does not have to rely on a specific sensor or group of sensors to determine analysis results. Additionally, Sensor Fusion 601 provides better performance due to machine learning algorithms stabilized by cross-referencing sensor information and removing faulty sensor inputs with information from a sensor cohort.

[0096] Figure 16 shows an example of the workflow of the sixth stage. Labels, lead times and corresponding metadata are consolidated from every single channel. The metadata includes geographic and temporal information. A map is created with channel locations and predicted labels from the previous stage for the entire environment at different points in time. A Bayesian model is built from Bayesian learning 603 using information obtained from other locations over multiple time periods. Labels across sensors update the previous ones with new labels and lead times. The footprint of the anomaly event is traced 604. The path of the newly emerged anomaly is used to identify the root cause.

[0097] Compared to the prior art, this step is novel because it integrates information across sensors through sensor fusion techniques to create a fault-tolerant model to address the rainforest environment and predict anomaly footprints by considering sensor data or known characteristics of the entire environment.

[0098] In rainforest, this step is to combine the labels from all guardians over a period of time to build a footprint traverse of the illegal logging life cycle. Figures 17-18 show the mapping of illegal logging fingerprints and footprint traversal according to one implementation.

[0099] In order to fuse the label data and the metadata, a link needs to be established between the predicted labels and the metadata.

[0100] Assume the following: Gi represents the geographic location for guardian i, i=1,2,...,n, Tj represents the timestamp y, j=1,2,...,m, Lij represents the predicted label or lead time in Gi of guardian at time Tj, for i = 1, 2, ..., n and j = 1, 2, ..., m.

[0101] The following connections need to be made: G1-T1-L11 G1-T2-L12 ... G1-Tm-L1m G2-T1-L21 G2-T2-L22 ... G2-Tm-L2m ... Gn-T1-Ln1 Gn-T2-Ln2 ... Gn-Tm-Lnm

[0102] Once the associations are established, static fingerprints can be created at 602. As in the example of Figure 17, there are four timestamps across the guardians. The legend is the same as in Figure 16: Black circle: Quiet stage Dashed circle: stage of anthropogenic disturbance Solid circle: chainsaw disturbance stage

[0103] As time progresses, the predicted labels also change. At T1, the labels of G1 and G2 change from the quiet stage to the disturbed stage, which may mean that some loggers have entered the forest. At T2, the loggers move in the forest and G2 and G3 are in the disturbed stage, while G1 goes back to the quiet stage. At T3, the loggers move further and start cutting trees in guardian G4. At T4, the cut area expands and G5 is also disturbed.

[0104] By fusing the geographic information, an acoustic fingerprint can be constructed. One result of this signature system is that the acoustic fingerprint identifies the temporal patterns of an acoustic signal based on its perceptual characteristics from a single channel.

[0105] On the other hand, independent fingerprints are insufficient in predicting the footprint of illegal logging activities due to two limitations: first, it is impossible to link the Guardian states over multiple time periods without knowing sequential information from the whole environment; second, if some sensors are unavailable, the prediction of that Guardian is also infeasible.

[0106] To solve these problems, a Bayesian network 603 is implemented to predict footprints. A Bayesian network is a probabilistic graphical model that represents a set of variables and their conditional dependencies through a directed acyclic graph (DAG). Bayesian networks are ideal for taking an event that has occurred and predicting the likelihood that several possible known causes were contributing factors. Bayesian network approaches include, but are not limited to, Dynamic Bayesian Networks (DBN) and Hidden Markov Models (HMM).

[0107] Therefore, when the state of a guardian is unavailable, the state of the indirect guardian can be used to supplement the state of the direct guardian. Using multiple sensors measuring the same quantity can increase robustness and fault tolerance. In addition, based on the sequential guardian states, the traversal patterns of illegal loggers can be identified.

[0108] In the example of FIG. 18, a footprint has been created and the probability of the G5 label can be predicted by fusing the labels G1 through G5 over multiple time periods.

[0109] The traversal of the dynamic footprint 604 is the crucial output of the system. The traversal pattern is learned by fusion data across the Guardian, taking into account its geographic and temporal metadata. When building the model, taking into account inputs from multiple radars, lidars and cameras as a priori knowledge, the traversal footprint provides the most comprehensive and therefore accurate model of the environment possible. This output can be used as a complete identification of the intruder's movement patterns to enhance investigations.

[0110] Another output of this system is an alert notification system. By setting a threshold for the length of anthropogenic disturbance identified for a particular guardian with a predicted footprint traverse, a reliable alert system can be created. For example, if G1 and G2 are experiencing significant amounts of anthropogenic disturbance, the guards can turn their attention to G4 and G5 to stop illegal loggers. The system can also avoid alert fatigue by adjusting the length of anthropogenic disturbance or the number of labels identified as stages of anthropogenic disturbance.

[0111] This dynamic footprint traversal can also be useful for optimizing ranger resources across guardians in rainforest locations. A vulnerability scale can be created for the guardians according to the predicted likelihood of illegal logging. In guardians with a high vulnerability scale, rangers can allocate more resources or deploy more guardians to monitor the area more thoroughly.

[0112] With implementations described herein, the dynamic acoustic signature system fuses data to predict fingerprints and footprints of illegal logging activities. The dynamic acoustic signature system uses stochastic processes, self-feedback error learning and sensor fusion processes to provide reliable and robust predictions. The system can thwart illegal logging by labeling acoustic signals and identifying stages of anthropogenic disturbance with limited human intervention.

[0113] The implementation can be applied to public utilities or forest rangers who want to estimate the footprint of loggers and stop illegal logging. In addition, stopping logging can extend to stopping fires. By replacing "chainsaw" with "fire" as the analysis target, the footprint of fires can also be learned and predicted. Furthermore, this target of the present invention can be extended from illegal loggers to tourists. Locating and rescuing lost tourists has a significant impact during a disaster.

[0114] In addition, since machine and product events also have signatures (fingerprints and footprints), the implementations described herein can be applied to industrial production with some minor modifications. For example, this approach can be used to identify "fault footprints in industrial engineering" to prevent faults in advance and to prevent root cause fault reasons. The assumption is that it should take time for a machine or product pattern to change from normal to abnormal, and this process can be learned by the system.

[0115] Implementations can be used to track down cascading faults: In the power grid example, a fault in a transformer can affect other transformers in a chain that must take over the load, affecting those who rely on power from that transformer.

[0116] 19 illustrates a system including multiple systems with connected sensors and a management device, according to one implementation. One or more sensor systems 1901-1, 1901-2, 1901-3, and 1901-4 are communicatively coupled to a network 1900 that is connected to a management device 1902 that facilitates functionality for an Internet of Things (IoT) gateway or other management system. The management device 1902 manages a database 1903 that includes historical data collected from the sensor systems 1901-1, 1901-2, 1901-3, and 1901-4, which may include labeled and unlabeled data received from the systems 1901-1, 1901-2, 1901-3, and 1901-4. In alternative implementations, data from sensor systems 1901-1, 1901-2, 1901-3, 1901-4 can be stored in a central repository or database, such as a proprietary database that captures data from an enterprise resource planning system, and management equipment 1902 can access or retrieve the data from the central repository or database. Such systems can include remote sensors connected to transmission devices to facilitate communication from the forest to equipment 1902, sensors attached to manufacturing systems, etc., depending on the desired implementation.

[0117] FIG. 20 illustrates an example computing environment that includes an example computing device suitable for use in some implementations, such as management device 1902 illustrated in FIG.

[0118] A computing device 2005 in the computing environment 2000 may include one or more processing units, cores or processors 2010, memory 2015 (e.g., RAM, ROM, etc.), internal storage 2020 (e.g., magnetic, optical, solid-state storage and / or organic), and / or I / O interface 2025, any of which may be coupled over a communication mechanism or bus 2030 to communicate information or may be embedded within the computing device 2005. The I / O interface 2025 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0119] The computing device 2005 may be communicatively coupled to an input / user interface 2035 and output devices / interfaces 2040. Either or both of the input / user interface 2035 and output devices / interfaces 2040 may be wired or wireless interfaces and may be removable. The input / user interface 2035 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor control, microphones, cameras, Braille, motion sensors, optical readers, etc.). The output devices / interfaces 2040 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some implementations, the input / user interface 2035 and output devices / interfaces 2040 may be embedded in the computing device 2005 or may be physically coupled to the computing device 2005. In other implementations, other computing devices may function as or provide the functionality of the input / user interface 2035 and the output device / interface 2040 for the computing device 2005 .

[0120] Examples of computing devices 2005 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios with one or more processors embedded and / or coupled thereto, etc.).

[0121] Computing device 2005 may be communicatively coupled (e.g., via I / O interface 2025) to external storage 2045 and network 2050 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 2005, or any connected computing device, may function as or provide services to or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or another label.

[0122] I / O interface 2025 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) to exchange information with at least all connected components, devices and networks in computing environment 2000. Network 2050 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0123] The computing device 2005 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media includes transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media includes magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage) and other non-volatile storage or memory.

[0124] In some examples of computing environments, the computer device 2005 may be used to implement techniques, methods, applications, processes, or computer-executable instructions. The computer-executable instructions may be retrieved from a transitory medium, stored on a non-transitory medium, and retrieved from the same. The executable instructions may be written in any programming, scripting, and machine language (e.g., C, C++, C#, Java (Registered Trademark) , Visual Basic (Registered Trademark) , Python (Registered Trademark) , Perl, JavaScript (Registered Trademark) etc.).

[0125] The processor 2010 can execute under any operating system (OS) (not shown) in a native or virtual environment. The logic unit 2060, the application programming interface (API) unit 2065, the input unit 2070, the output unit 2075 and the various units can implement one or more applications including inter-unit communication mechanisms 2095 for communicating with each other, with the OS and with other applications (not shown). The described units and elements can be modified in terms of design, function, configuration or implementation and are not limited to the description given.

[0126] In some implementations, when information or instructions to execute are received by the API unit 2065, it may be communicated to one or more other units (e.g., logic unit 2060, input unit 2070, output unit 2075). In some implementations described above, logic unit 2060 may be configured to control information flow between units and direct services provided by API unit 2065, input unit 2070, output unit 2075. For example, the flow of one or more processes or implementations may be controlled by logic unit 2060 alone or in combination with API unit 2065. Input unit 2070 may be configured to obtain inputs for calculations described in the implementations, and output unit 2075 may be configured to provide outputs based on the calculations described in the implementations.

[0127] The processor 2010 may be configured, for example, to identify and label patterns based on characteristics of sensor data streamed from one or more sensors in the network of sensors, as shown in Figures 2, 6 and 8, process the identified and labeled patterns to estimate the probability and lead time for a change in an area associated with one or more sensors from a current sensor stage to another sensor stage, as shown in Figures 13-15, including a sequential error learning process configured to improve accuracy, and determine, for neighboring sensors to one or more sensors in the network of sensors, the probability of a change in sensor stage of other areas associated with the neighboring sensors, as shown in Figure 16.

[0128] The processor 2010 may be configured to identify and label patterns based on characteristics of the sensor data streamed from one or more sensors in the network by transforming acoustic data in the streamed sensor data from the time domain to the frequency domain, deriving frequency domain features from the transformed acoustic signal, applying dimensionality reduction to the frequency domain features, clustering the dimensionality reduced frequency domain features, and applying fingerprint analysis to identify sensor stages, as shown in Figures 6-8.

[0129] The processor 2010 may be configured to apply fingerprint analysis to identify the signal fingerprint from background noise, identify leading indicators of anomalous events, and identify sensor stages by analyzing event associations between anomalies and their sequences through frequency band distribution and image classification, as shown in Figures 8-12.

[0130] The processor 2010 may be configured to identify and label patterns based on characteristics of the sensor data streamed from one or more sensors in the network of sensors by synthesizing data to enhance the sensor data, as shown in FIG. 4, including extracting signal features from the separated noise and target signals to replicate the anomalous and noise signals, combining the replicated anomalous and noise signals through superposition to form synthetic data, and validating the synthetic data.

[0131] The processor 2010 may be configured to identify and label patterns based on characteristics of the sensor data streamed from one or more sensors in the network of sensors by performing outlier removal on the sensor data based on audio and image metadata learning, as shown in FIG.

[0132] The processor 2010 may be configured to process the identified and labeled patterns to estimate probabilities and lead times for changes in an area associated with one or more sensors from a current stage to another stage through a probabilistic process trained on the temporal patterns, as shown in Figures 13-14, and output probabilities and lead times for all types of labeled sensor stages.

[0133] Depending on the desired implementation, the sequential error learning process configured to improve accuracy may include a self-feedback loop configured to learn the error inherited from the stochastic process and improve the pseudo labels between training phases.

[0134] The processor 2010 may be configured to determine the probability of a change in sensor stage in other areas relative to the neighboring sensors based on a Bayesian learning model configured to determine the probability and estimated occurrence time of a change in sensor stage in other areas based on fused fingerprint labels with geographical and temporal information from neighboring sensors in the network of sensors, as shown in Figures 16 and 17.

[0135] The processor 2010 is configured to determine the probability of a sensor stage change in other areas relative to the neighboring sensors by generating a dynamic footprint including sensors of the network of sensors having a probability of a sensor stage change for neighboring sensors to one or more sensors in the network of sensors, the dynamic footprint providing a visualization of sensors of the network of sensors undergoing a state change over time in a chronological order, as shown in FIG. 18 .

[0136] Depending on the desired implementation, as shown in FIG. 16, the sensor stage (e.g., current sensor stage, another sensor stage) can be one of anthropogenic disturbance, dense chainsaw, light chainsaw, or quiet time.

[0137] Implementations can also be extended to determine other state changes depending on the desired implementation (e.g., as applied to a factory setting or other IoT setting). In such implementations, the processor 2010 can be configured to detect a first event from sensor data streamed from one or more sensors in the network of sensors, process the first event to determine a probability of a second event for an area associated with the one or more sensors and an estimated lead time of the second event, and determine for neighboring sensors to the one or more sensors in the network of sensors a probability of a state change to the first event or the second event for other areas associated with the neighboring sensors. The state change can be any type of fault condition depending on the desired implementation, and the state change for the neighboring sensors can be utilized to detect cascading fault events.

[0138] In one example of such an implementation for detecting other state changes to locate cascading faults, an example may include an electrical grid having multiple transformers. When a transformer is fully loaded and fails, other adjacent transformers must handle the additional load to assist the grid. However, if that load also causes a fault in an adjacent transformer, a cascading fault event may occur where a fault in one transformer in turn causes a fault in other adjacent transformers.

[0139] Thus, implementations can be used to determine if there is an early cascading failure in the fingerprint analysis, from which the system can be shut down immediately to prevent the cascading failure.

[0140] Some portions of the detailed description have been presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In one implementation, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0141] Unless otherwise specified, and as will be apparent from the description, the description utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like throughout the description will be understood to include the actions and processes of a computer system or other information processing device that manipulates data represented as physical (electronic) quantities in the registers and memory of the computer system and converts it into other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display device.

[0142] An implementation example may also relate to an apparatus for performing the operations of the present specification. The apparatus may be specially constructed for the required purpose or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. A computer-readable storage medium may include, but is not limited to, tangible media such as optical disks, magnetic disks, read-only memories, random access memories, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. A computer-readable signal medium may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to a particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions to perform the operations of a desired implementation.

[0143] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Additionally, the implementations are not described with reference to a particular programming language. It will be appreciated that a variety of programming languages ​​can be used to implement the techniques of the implementations described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0144] As known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. While various aspects of the implementations can be implemented using circuits and logic (hardware), other aspects can be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform a method for performing the implementation of the present application. Furthermore, some implementations of the present application can be performed exclusively in hardware, while other implementations can be performed exclusively in software. Furthermore, the various functions described can be performed in a single unit or distributed among several components in any number of ways. When performed by software, the method can be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If necessary, the instructions can be stored on the medium in compressed and / or encrypted format.

[0145] Moreover, other implementations of the present application will be apparent to those skilled in the art upon consideration of the specification and practice of the techniques of the present application. Various aspects and / or components of the described implementations may be used alone or in any combination. It is intended that the specification and implementations be considered as examples only, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. A method of operating a system, the method comprising: Identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors; learning the identified and labeled patterns to estimate a probability and a lead time indicative of an elapsed time for a change in a region associated with the one or more sensors from a current sensor data state to another sensor data state, the learning including a sequential error learning process configured to learn about an error in a label attached to the pattern if an error rate of the label is equal to or greater than a threshold value to improve the accuracy of the estimation of the probability and the lead time; and determining, for neighboring sensors adjacent to the one or more sensors in the network of sensors, a probability of a change in state of sensor data for other regions associated with the neighboring sensors; The method includes:

2. The identifying and labeling of the patterns based on the characteristics of the sensor data streamed from the one or more sensors in the network includes: converting acoustic data in the streamed sensor data from a time domain to a frequency domain; deriving frequency domain features from the transformed acoustic signal; applying dimensionality reduction to the frequency domain features; clustering the reduced-dimensionality frequency domain features; and applying a fingerprint analysis to identify temporal patterns in the acoustic signal to identify a state of the sensor data; The method of claim 1 , comprising:

3. The applying of the fingerprint analysis to identify a state of the sensor data includes: identifying a fingerprint of a signal from background noise contained in said audio signal; Identifying leading indicators that occur before an anomalous event; and Analyzing event associations between the anomalous events and their sequences through frequency band distribution and image classification. The method of claim 2 , comprising:

4. The identifying and labeling patterns based on the characteristics of the sensor data streamed from the one or more sensors in the network of sensors further includes synthesizing data to augment the sensor data, the synthesizing data comprising: Extracting signal features from noise signals and target signals contained in the acoustic signal to replicate the abnormal abnormal signals and the noise signals in a desired environment; combining the replicated abnormal signals and the noise signals through superposition to form synthetic data; and Verifying that the synthetic data is indistinguishable from the acoustic signal. The method of claim 1 , comprising:

5. 2. The method of claim 1 , wherein the processing of the identified and labeled patterns to estimate the probability and the lead time for a change in the region associated with the one or more sensors from the current sensor data state to another sensor data state is done through a probabilistic process trained on temporal patterns to output the probability and the lead time for all types of labeled sensor data states.

6. The method of claim 1, wherein the sequential error learning process configured to improve the estimation accuracy includes a self-feedback loop configured to learn errors inherited from a stochastic process for learning the patterns and to improve pseudo-labels for acoustic signals in the streamed sensor data between learning phases.

7. 2. The method of claim 1, wherein for the neighboring sensors to the one or more sensors in the network of sensors, determining the probability of a change in state of the sensor data of other regions associated with the neighboring sensors is based on a Bayesian learning model configured to determine the probability and estimated occurrence time of a change in state of the sensor data of the other regions based on metadata including geographical and temporal information from the neighboring sensors in the network of sensors.

8. 2. The method of claim 1, wherein determining, for the neighboring sensors to the one or more sensors in the network of sensors, the probability of a change in state of the sensor data of the other region associated with the neighboring sensors further comprises generating a dynamic footprint including sensors of the network of sensors having the probability of a change in state of the sensor data, the dynamic footprint providing a visualization of the sensors of the network of sensors undergoing a change in state of the sensor data over time in chronological order.

9. The method of claim 1 , wherein each of the current sensor data state and the another sensor data state is one of an anthropogenic disturbance, a heavy chainsaw, a light chainsaw, or a quiet time.

10. A non-transitory computer readable medium storing instructions for carrying out a process, the instructions comprising: Identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors; learning the identified and labeled patterns to estimate a probability and a lead time indicative of an elapsed time for a change in a region associated with the one or more sensors from a current sensor data state to another sensor data state, the learning including a sequential error learning process configured to learn about an error in a label attached to the pattern if an error rate of the label is equal to or greater than a threshold value to improve the accuracy of the estimation of the probability and the lead time; and determining, for neighboring sensors adjacent to the one or more sensors in the network of sensors, a probability of a change in state of sensor data for other regions associated with the neighboring sensors; A non-transitory computer readable medium comprising:

11. Identifying and labeling patterns based on characteristics of sensor data streamed from one or more sensors in a network of sensors; learning the identified and labeled patterns to estimate a probability and a lead time indicative of an elapsed time for a change in a region associated with the one or more sensors from a current sensor data state to another sensor data state, the learning including a sequential error learning process configured to learn about an error in a label attached to the pattern if an error rate of the label is equal to or greater than a threshold value to improve the accuracy of the estimation of the probability and the lead time; and determining, for neighboring sensors adjacent to the one or more sensors in the network of sensors, a probability of a change in state of sensor data for other regions associated with the neighboring sensors; a processor configured to: Including equipment.

Images